diff --git a/README.md b/README.md index 4ee38a0..fb1e849 100644 --- a/README.md +++ b/README.md @@ -66,7 +66,7 @@ or third party libraries (such as [Eris](https://github.com/abalabahaha/eris)). Requirements: - **Python 3.7+** -- PostgreSQL (tested using 9.6+) +- PostgreSQL (tested using 9.6+), SQL knowledge is recommended. - gifsicle for GIF emoji and avatar handling - [pipenv] @@ -133,6 +133,9 @@ You can check if your instance is running by performing an HTTP `GET` request on the `/api/v6/gateway` endpoint. For basic websocket testing, a tool such as [ws](https://github.com/hashrocket/ws) can be used. +After checking that it actually works, `docs/operating.md` continues on common +operations for a Litecord instance. + ## Updating Update the code and run any new database migrations: diff --git a/docs/README.md b/docs/README.md index 7476b25..2228ab9 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,3 +1,5 @@ # Internal documentation -The Litecord Voice Server Protocol (LVSP) is documented here. + - `admin_api.md` for Litecord's Admin API. + - `lvsp.md` for the Litecord Voice Server Protocol. + - `pubsub.md` for how Pub/Sub works in Litecord. diff --git a/docs/operating.md b/docs/operating.md new file mode 100644 index 0000000..b30fe3f --- /dev/null +++ b/docs/operating.md @@ -0,0 +1,34 @@ +# Operating a Litecord instance + +`./manage.py` contains common admin tasks that you may want to do to the +instance, e.g make someone an admin, or migrate the database, etc. + +Note, however, that many commands (example the ones related to user deletion) +may be moved to the Admin API without proper notice. There is no frontend yet +for the Admin API. + +The possible actions on `./manage.py` can be accessed via `./manage.py -h`, or +`pipenv run ./manage.py -h` if you're on pipenv. + +## `./manage.py generate_token`? + +You can generate a user token but only if that user is a bot. + +## Instance invites + +If your instance has registrations disabled you can still get users to the +instance via instance invites. This is something only Litecord does, using a +separate API endpoint. + +Use `./manage.py makeinv` to generate an instance invite, give it out to users, +point them towards `https:///invite_register.html`. Things +should be straightforward from there. + +## Making someone Staff + +**CAUTION:** Making someone staff, other than giving the Staff badge on their +user flags, also gives complete access over the Admin API. Only make staff the +people you (the instance OP) can trust. + +Use the `./manage.py make_staff` management task to make someone staff. There is +no way to remove someone's staff with a `./manage.py` command *yet.* diff --git a/docs/pubsub.md b/docs/pubsub.md new file mode 100644 index 0000000..66dc0fe --- /dev/null +++ b/docs/pubsub.md @@ -0,0 +1,66 @@ +# Pub/Sub (Publish/Subscribe) + +Please look over wikipedia or other sources to understand what it is. +This only documents how an events are generated and dispatched to clients. + +## Event definition + +Events are composed of two things: + - Event type + - Event payload + +More information on how events are structured are in the Discord Gateway +API documentation. + +## `StateManager` (litecord.gateway.state\_manager) + +StateManager stores all available instances of `GatewayState` that identified. +Specific information is over the class' docstring, but we at least define +what it does here for the next class: + +## `EventDispatcher` (litecord.dispatcher) + +EventDispatcher declares the main interface between clients and the side-effects +(events) from topics they can subscribe to. + +The topic / channel in EventDispatcher can be a User, or a Guild, or a Channel, +etc. Users subscribe to the channels they have access to manually, and get +unsubscribed when they e.g leave the guild the channel is on, etc. + +Channels are identified by their backend and a given key uniquely identifying +the channel. The backend can be `guild`, `member`, `channel`, `user`, +`friend`, and `lazy_guild`. Backends *can* store the list of subscribers, but +that is not required. + +Each backend has specific logic around dispatching a single event towards +all the subscribers of the given key. For example, the `guild` backend only +dispatches events to shards that are properly subscribed to the guild, +instead of all shards. The `user` backend just dispatches the event to +all available shards in `StateManager`, etc. + +EventDispatcher also implements common logic, such as `dispatch_many` to +dispatch a single event to multpiple keys in a single backend. This is useful +e.g a user is updated and you want to dispatch `USER_UPDATE` events to many +guilds without having to write a loop yourself. + +## Backend superclasses (litecord.pubsub.dispatcher) + +The backend superclasses define what methods backends must provide to be +fully functional within EventDispatcher. They define e.g what is the type +of the keys, and have some specific backend helper methods, such as +`_dispatch_states` to dispatch an event to a list of states without +worrying about errors or writing the loop. + +The other available superclass is `DispatchWithState` for backends that +require a list of subscribers to not repeat code. The only required method +to be implemented is `dispatch()` and you can see how that works out +on the backends that inherit from this class. + +## Sending an event, practical + +Call `app.dispatcher.dispatch(backend_string, key, event_type, event_payload)`. + +example: + - `dispatch('guild', guild_id, 'GUILD_UPDATE', guild)`, and other backends. + The rules on how each backend dispatches its events can be found on the + specific backend class. diff --git a/docs/structure.md b/docs/structure.md new file mode 100644 index 0000000..f5e0fea --- /dev/null +++ b/docs/structure.md @@ -0,0 +1,42 @@ +# Project structure + +## `attachments` and `images` + +They're empty folders on purpose. Litecord will write files to them to hold +message attachments or avatars. + +## `manage` + +Contains the `manage.py` script's main function, plus all the commands. +A point of interest is the `manage/cmd/migration/scripts` folder, as they hold +all the SQL scripts required for migrations. + +## `litecord` + +The folder + `run.py` contain all of the backend's source code. The backend runs +Quart as its HTTP server, and a `websockets` server for the Gateway. + + - `litecord/blueprints` for everything HTTP related. + - `litecord/gateway` for main things related to the websocket or the Gateway. + - `litecord/embed` contains code related to embed sanitization, schemas, and + mediaproxy contact. + - `litecord/ratelimits` hold the ratelimit implementation copied from + discord.py plus a simple manager to hold the ratelimit objects. a point of + interest is `litecord/ratelimits/handler.py` that holds the main thing. + - `litecord/pubsub` is defined on `docs/pubsub.md`. + - `litecord/voice` holds the voice implementation, LVSP client, etc. + +There are other files around `litecord/`, e.g the snowflake library, presence/ +image/job managers, etc. + +## `static` + +Holds static files, such as a basic index page and the `invite_register.html` +page. + +## `tests` + +Tests are run with `pytest` and the asyncio plugin for proper testing. A point +of interest is `tests/conftest.py` as it contains test-specific configuration +for the app object. Adding a test is trivial, as pytest will match against any +file containing `test_` as a prefix. diff --git a/litecord/auth.py b/litecord/auth.py index 715865f..b52e1a1 100644 --- a/litecord/auth.py +++ b/litecord/auth.py @@ -20,6 +20,7 @@ along with this program. If not, see . import base64 import binascii from random import randint +from typing import Tuple import bcrypt from asyncpg import UniqueViolationError @@ -35,7 +36,21 @@ from litecord.enums import UserFlags log = Logger(__name__) -async def raw_token_check(token, db=None): +async def raw_token_check(token: str, db=None) -> int: + """Check if a given token is valid. + + Returns + ------- + int + The User ID of the given token. + + Raises + ------ + Unauthorized + If token is not properly formatted, or if the user does not exist. + Forbidden + If token validation fails. + """ db = db or app.db # just try by fragments instead of @@ -80,7 +95,7 @@ async def raw_token_check(token, db=None): raise Forbidden('Invalid token') -async def token_check(): +async def token_check() -> int: """Check token information.""" # first, check if the request info already has a uid try: @@ -101,7 +116,7 @@ async def token_check(): return user_id -async def admin_check(): +async def admin_check() -> int: """Check if the user is an admin.""" user_id = await token_check() @@ -148,13 +163,19 @@ async def check_username_usage(username: str, db=None): async def create_user(username: str, email: str, password: str, - db=None, loop=None): - """Create a single user.""" + db=None, loop=None) -> Tuple[int, str]: + """Create a single user. + + Generates a distriminator and other information. You can fetch the user + data back with :meth:`Storage.get_user`. + """ db = db or app.db loop = loop or app.loop new_id = get_snowflake() + # TODO: unified discrim generation based off username, that also includes + # the check_username_usage() new_discrim = randint(1, 9999) new_discrim = '%04d' % new_discrim @@ -164,9 +185,10 @@ async def create_user(username: str, email: str, password: str, try: await db.execute(""" - INSERT INTO users (id, email, username, - discriminator, password_hash) - VALUES ($1, $2, $3, $4, $5) + INSERT INTO users + (id, email, username, discriminator, password_hash) + VALUES + ($1, $2, $3, $4, $5) """, new_id, email, username, new_discrim, pwd_hash) except UniqueViolationError: raise BadRequest('Email already used.') diff --git a/litecord/gateway/gateway.py b/litecord/gateway/gateway.py index d6ec58a..f435d16 100644 --- a/litecord/gateway/gateway.py +++ b/litecord/gateway/gateway.py @@ -22,9 +22,8 @@ from litecord.gateway.websocket import GatewayWebsocket async def websocket_handler(app, ws, url): - """Main websocket handler, checks query arguments - when connecting to the gateway and spawns a - GatewayWebsocket instance for the connection.""" + """Main websocket handler, checks query arguments when connecting to + the gateway and spawns a GatewayWebsocket instance for the connection.""" args = urllib.parse.parse_qs( urllib.parse.urlparse(url).query ) @@ -54,6 +53,9 @@ async def websocket_handler(app, ws, url): if gw_compress and gw_compress not in ('zlib-stream',): return await ws.close(1000, 'Invalid gateway compress') - gws = GatewayWebsocket(ws, app, v=gw_version, - encoding=gw_encoding, compress=gw_compress) + gws = GatewayWebsocket( + ws, app, v=gw_version, encoding=gw_encoding, compress=gw_compress) + + # this can be run with a single await since this whole coroutine + # is already running in the background. await gws.run() diff --git a/litecord/gateway/websocket.py b/litecord/gateway/websocket.py index 4cc22aa..436001a 100644 --- a/litecord/gateway/websocket.py +++ b/litecord/gateway/websocket.py @@ -65,30 +65,34 @@ WebsocketObjects = collections.namedtuple( def encode_json(payload) -> str: + """Encode a given payload to JSON.""" return json.dumps(payload, separators=(',', ':'), cls=LitecordJSONEncoder) def decode_json(data: str): + """Decode from JSON.""" return json.loads(data) def encode_etf(payload) -> str: - # The thing with encoding ETF is that with json we have LitecordJSONEncoder - # which takes care of converting e.g datetime objects to their ISO - # representation. + """Encode a payload to ETF (External Term Format). - # so, to keep things working, i'll to a json pass on the payload, then send - # the decoded payload back to earl. + This gives a JSON pass on the given payload (via calling encode_json and + then decode_json) because we may want to encode objects that can only be + encoded by LitecordJSONEncoder. + Earl-ETF does not give the same interface for extensibility, hence why we + do the pass. + """ sanitized = encode_json(payload) sanitized = decode_json(sanitized) return earl.pack(sanitized) def _etf_decode_dict(data): - # NOTE: this is a very slow implementation to - # decode the dictionary. + """Decode a given dictionary.""" + # NOTE: this is very slow. if isinstance(data, bytes): return data.decode() @@ -109,6 +113,7 @@ def _etf_decode_dict(data): return result def decode_etf(data: bytes): + """Decode data in ETF to any.""" res = earl.unpack(data) if isinstance(res, bytes): @@ -269,7 +274,7 @@ class GatewayWebsocket: task_wrapper('hb wait', self._hb_wait(interval)) ) - async def send_hello(self): + async def _send_hello(self): """Send the OP 10 Hello packet over the websocket.""" # random heartbeat intervals interval = randint(40, 46) * 1000 @@ -367,7 +372,7 @@ class GatewayWebsocket: 'friend_suggestion_count': 0, - + # those are unused default values. 'connected_accounts': [], 'experiments': [], 'guild_experiments': [], @@ -387,7 +392,7 @@ class GatewayWebsocket: uready = await self._user_ready() private_channels = ( - await self.user_storage.get_dms(user_id) + + await self.user_storage.get_dms(user_id) + await self.user_storage.get_gdms(user_id) ) @@ -406,6 +411,8 @@ class GatewayWebsocket: self.ext.loop.create_task(self._guild_dispatch(guilds)) async def _check_shards(self, shard, user_id): + """Check if the given `shard` value in IDENTIFY has good enough values. + """ current_shard, shard_count = shard guilds = await self.ext.db.fetchval(""" @@ -612,10 +619,6 @@ class GatewayWebsocket: # setting new presence to state await self.update_status(presence) - def voice_key(self, channel_id: int, guild_id: int): - """Voice state key.""" - return (self.state.user_id, self.state.session_id) - async def _vsu_get_prop(self, state, data): """Get voice state properties from data, fallbacking to user settings.""" @@ -837,6 +840,11 @@ class GatewayWebsocket: await self._req_guild_members(gid, [], query, limit) async def _guild_sync(self, guild_id: int): + """Synchronize a guild. + + Fetches the members and presences of a guild and dispatches a + GUILD_SYNC event with that info. + """ members = await self.storage.get_member_data(guild_id) member_ids = [int(m['user']['id']) for m in members] @@ -979,7 +987,7 @@ class GatewayWebsocket: self.state.session_id, ranges ) - async def process_message(self, payload): + async def _process_message(self, payload): """Process a single message coming in from the client.""" try: op_code = payload['op'] @@ -998,7 +1006,7 @@ class GatewayWebsocket: if self._check_ratelimit('messages', self.state.session_id): raise WebsocketClose(4008, 'You are being ratelimited.') - async def listen_messages(self): + async def _listen_messages(self): """Listen for messages coming in from the websocket.""" # close anyone trying to login while the @@ -1018,9 +1026,11 @@ class GatewayWebsocket: await self._msg_ratelimit() payload = self.decoder(message) - await self.process_message(payload) + await self._process_message(payload) def _cleanup(self): + """Cleanup any leftover tasks, and remove the connection from the + state manager.""" for task in self.wsp.tasks.values(): task.cancel() @@ -1058,11 +1068,11 @@ class GatewayWebsocket: ) async def run(self): - """Wrap listen_messages inside + """Wrap :meth:`listen_messages` inside a try/except block for WebsocketClose handling.""" try: - await self.send_hello() - await self.listen_messages() + await self._send_hello() + await self._listen_messages() except websockets.exceptions.ConnectionClosed as err: log.warning('conn close, state={}, err={}', self.state, err) except WebsocketClose as err: diff --git a/litecord/jobs.py b/litecord/jobs.py index 55b8f1d..b27099e 100644 --- a/litecord/jobs.py +++ b/litecord/jobs.py @@ -24,18 +24,27 @@ log = Logger(__name__) class JobManager: - """Manage background jobs""" + """Background job manager. + + Handles closing all existing jobs when going on a shutdown. This does not + use helpers such as asyncio.gather and asyncio.Task.all_tasks. It only uses + its own internal list of jobs. + """ def __init__(self, loop=None): self.loop = loop or asyncio.get_event_loop() self.jobs = [] async def _wrapper(self, coro): + """Wrapper coroutine for other coroutines. This adds a simple + try/except for general exceptions to be logged. + """ try: await coro except Exception: log.exception('Error while running job') def spawn(self, coro): + """Spawn a given future or coroutine in the background.""" task = self.loop.create_task( self._wrapper(coro) ) @@ -43,5 +52,10 @@ class JobManager: self.jobs.append(task) def close(self): + """Close the job manager, cancelling all existing jobs. + + It is the job's responsibility to handle the given CancelledError + and release any acquired resources. + """ for job in self.jobs: job.cancel() diff --git a/litecord/permissions.py b/litecord/permissions.py index 4b062e8..d0ebb75 100644 --- a/litecord/permissions.py +++ b/litecord/permissions.py @@ -64,6 +64,14 @@ class _RawPermsBits(ctypes.LittleEndianStructure): class Permissions(ctypes.Union): + """Main permissions class. Holds helper functions to convert between + the bitfield and an integer, etc. + + Parameters + ---------- + val + The permissions value as an integer. + """ _fields_ = [ ('bits', _RawPermsBits), ('binary', ctypes.c_uint64), @@ -78,9 +86,6 @@ class Permissions(ctypes.Union): def __int__(self): return self.binary - def numby(self): - return self.binary - ALL_PERMISSIONS = Permissions(0b01111111111101111111110111111111) @@ -152,6 +157,7 @@ async def base_permissions(member_id, guild_id, storage=None) -> Permissions: def overwrite_mix(perms: Permissions, overwrite: dict) -> Permissions: + """Mix a single permission with a single overwrite.""" # we make a copy of the binary representation # so we don't modify the old perms in-place # which could be an unwanted side-effect @@ -168,6 +174,24 @@ def overwrite_mix(perms: Permissions, overwrite: dict) -> Permissions: def overwrite_find_mix(perms: Permissions, overwrites: dict, target_id: int) -> Permissions: + """Mix a given permission with a given overwrite. + + Returns the given permission if an overwrite is not found. + + Parameters + ---------- + perms + The permissions for the given target. + overwrites + The overwrites for the given actor (mostly channel). + target_id + The target's ID in the overwrites dict. + + Returns + ------- + Permissions + The mixed permissions object. + """ overwrite = overwrites.get(target_id) if overwrite: @@ -251,8 +275,9 @@ async def compute_overwrites(base_perms: Permissions, return perms -async def get_permissions(member_id, channel_id, *, storage=None): - """Get all the permissions for a user in a channel.""" +async def get_permissions(member_id: int, channel_id, + *, storage=None) -> Permissions: + """Get the permissions for a user in a channel.""" if not storage: storage = app.storage @@ -264,5 +289,5 @@ async def get_permissions(member_id, channel_id, *, storage=None): base_perms = await base_permissions(member_id, guild_id, storage) - return await compute_overwrites(base_perms, member_id, - channel_id, guild_id, storage) + return await compute_overwrites( + base_perms, member_id, channel_id, guild_id, storage) diff --git a/litecord/presence.py b/litecord/presence.py index bf7bad9..2c2b6a5 100644 --- a/litecord/presence.py +++ b/litecord/presence.py @@ -68,6 +68,12 @@ def _best_presence(shards): async def _pres(storage, user_id: int, status_obj: dict) -> dict: + """Convert a given status into a presence, given the User ID and the + :class:`Storage` instance. + + This adds the required `presences` array, that isn't used, due to Litecord's + lack of Rich Presence. + """ ext = { 'user': await storage.get_user(user_id), 'activities': [], @@ -77,7 +83,11 @@ async def _pres(storage, user_id: int, status_obj: dict) -> dict: class PresenceManager: - """Presence related functions.""" + """Presence management. + + Has common functions to deal with fetching or updating presences, including + side-effects (events). + """ def __init__(self, app): self.storage = app.storage self.user_storage = app.user_storage @@ -159,7 +169,9 @@ class PresenceManager: 'activities': [game] if game else [] } - def _sane_session(session_id): + # given a session id, return if the session id actually connects to + # a given user, and if the state has not been dispatched via lazy guild. + def _session_check(session_id): state = self.state_manager.fetch_raw(session_id) uid = int(member['user']['id']) @@ -175,8 +187,7 @@ class PresenceManager: # gets a PRESENCE_UPDATE await self.dispatcher.dispatch_filter( 'guild', guild_id, - _sane_session, - + _session_check, 'PRESENCE_UPDATE', pres_update_payload ) diff --git a/litecord/schemas.py b/litecord/schemas.py index 855673a..c006288 100644 --- a/litecord/schemas.py +++ b/litecord/schemas.py @@ -49,7 +49,8 @@ EMOJO_MENTION = re.compile(r'<:(\.+):(\d+)>', re.A | re.M) ANIMOJI_MENTION = re.compile(r'', re.A | re.M) -def _in_enum(enum, value: int): +def _in_enum(enum, value) -> bool: + """Return if a given value is in the enum.""" try: enum(value) return True @@ -58,6 +59,7 @@ def _in_enum(enum, value: int): class LitecordValidator(Validator): + """Main validator class for Litecord, containing custom types.""" def _validate_type_username(self, value: str) -> bool: """Validate against the username regex.""" return bool(USERNAME_REGEX.match(value)) @@ -88,6 +90,10 @@ class LitecordValidator(Validator): return False def _validate_type_voice_region(self, value: str) -> bool: + # TODO: call voice manager for regions instead of hardcoding + + # I'm sure the context would be there at least in a basic level, so + # we can access the app. return value.lower() in ('brazil', 'us-east', 'us-west', 'us-south', 'russia') @@ -118,6 +124,7 @@ class LitecordValidator(Validator): except (TypeError, ValueError): return False + # nobody is allowed to use the INCOMING and OUTGOING rel types return val in (RelationshipType.FRIEND.value, RelationshipType.BLOCK.value) @@ -148,8 +155,18 @@ class LitecordValidator(Validator): def validate(reqjson: Union[Dict, List], schema: Dict, raise_err: bool = True) -> Dict: - """Validate a given document (user-input) and give - the correct document as a result. + """Validate the given user-given data against a schema, giving the + "correct" version of the document, with all defaults applied. + + Parameters + ---------- + reqjson: + The input data + schema: + The schema to validate reqjson against + raise_err: + If we should raise a BadRequest error when the validation + fails. Default is true. """ validator = LitecordValidator(schema) diff --git a/litecord/snowflake.py b/litecord/snowflake.py index d121f7a..fd85995 100644 --- a/litecord/snowflake.py +++ b/litecord/snowflake.py @@ -24,10 +24,7 @@ snowflake.py - snowflake helper functions File brought in from litecord-reference(https://github.com/lnmds/litecord-reference) """ -import os import time -import base64 -import hashlib import datetime # encoded in ms @@ -41,14 +38,6 @@ WORKER_ID = 1 Snowflake = int -def get_invite_code() -> str: - """Get a random invite code.""" - random_stuff = hashlib.sha512(os.urandom(1024)).digest() - code = base64.urlsafe_b64encode(random_stuff).decode().replace('=', '5') \ - .replace('_', 'W').replace('-', 'm') - return code[:6] - - def _snowflake(timestamp: int) -> Snowflake: """Get a snowflake from a specific timestamp diff --git a/litecord/system_messages.py b/litecord/system_messages.py index 0f13b8e..a9e1cab 100644 --- a/litecord/system_messages.py +++ b/litecord/system_messages.py @@ -133,7 +133,26 @@ async def _handle_gdm_icon_edit(app, channel_id, author_id): async def send_sys_message(app, channel_id: int, m_type: MessageType, *args, **kwargs) -> int: - """Send a system message.""" + """Send a system message. + + The handler for a given message type MUST return an integer, that integer + being the message ID generated. This function takes care of feching the + message and dispatching the needed event. + + Parameters + ---------- + app + The app instance. + channel_id + The channel ID to send the system message to. + m_type + The system message's type. + + Returns + ------- + int + The message ID. + """ try: handler = { MessageType.CHANNEL_PINNED_MESSAGE: _handle_pin_msg, diff --git a/litecord/utils.py b/litecord/utils.py index d19df06..db34b28 100644 --- a/litecord/utils.py +++ b/litecord/utils.py @@ -39,6 +39,7 @@ async def async_map(function, iterable: Iterable) -> list: async def task_wrapper(name: str, coro): + """Wrap a given coroutine in a task.""" try: await coro except asyncio.CancelledError: @@ -136,7 +137,10 @@ def mmh3(inp_str: str, seed: int = 0): class LitecordJSONEncoder(JSONEncoder): + """Custom JSON encoder for Litecord.""" def default(self, value: Any): + """By default, this will try to get the to_json attribute of a given + value being JSON encoded.""" try: return value.to_json except AttributeError: @@ -144,8 +148,7 @@ class LitecordJSONEncoder(JSONEncoder): async def pg_set_json(con): - """Set JSON and JSONB codecs for an - asyncpg connection.""" + """Set JSON and JSONB codecs for an asyncpg connection.""" await con.set_type_codec( 'json', encoder=lambda v: json.dumps(v, cls=LitecordJSONEncoder),