From a9be2a7cce2d999471ad1d3142076abf1bf6dbc6 Mon Sep 17 00:00:00 2001 From: Luna Date: Wed, 20 Mar 2019 00:35:30 -0300 Subject: [PATCH 01/15] docs: add docs/pubsub.md, revamp README --- docs/README.md | 4 +++- docs/pubsub.md | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+), 1 deletion(-) create mode 100644 docs/pubsub.md diff --git a/docs/README.md b/docs/README.md index 7476b25..2228ab9 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,3 +1,5 @@ # Internal documentation -The Litecord Voice Server Protocol (LVSP) is documented here. + - `admin_api.md` for Litecord's Admin API. + - `lvsp.md` for the Litecord Voice Server Protocol. + - `pubsub.md` for how Pub/Sub works in Litecord. diff --git a/docs/pubsub.md b/docs/pubsub.md new file mode 100644 index 0000000..b8effc8 --- /dev/null +++ b/docs/pubsub.md @@ -0,0 +1,57 @@ +# Pub/Sub (Publish/Subscribe) + +Please look over wikipedia or other sources to understand what it is. +This only documents how an events are generated and dispatched to clients. + +## Event definition + +Events are composed of two things: + - Event type + - Event payload + +More information on how events are structured are in the Discord Gateway +API documentation. + +## `StateManager` (litecord.gateway.state\_manager) + +StateManager stores all available instances of `GatewayState` that identified. +Specific information is over the class' docstring, but we at least define +what it does here for the next class: + +## `EventDispatcher` (litecord.dispatcher) + +EventDispatcher declares the main interface between clients and the side-effects +(events) from topics they can subscribe to. + +The topic / channel in EventDispatcher can be a User, or a Guild, or a Channel, +etc. Users subscribe to the channels they have access to manually, and get +unsubscribed when they e.g leave the guild the channel is on, etc. + +Channels are identified by their backend and a given key uniquely identifying +the channel. The backend can be `guild`, `member`, `channel`, `user`, +`friend`, and `lazy_guild`. Backends *can* store the list of subscribers, but +that is not required. + +Each backend has specific logic around dispatching a single event towards +all the subscribers of the given key. For example, the `guild` backend only +dispatches events to shards that are properly subscribed to the guild, +instead of all shards. The `user` backend just dispatches the event to +all available shards in `StateManager`, etc. + +EventDispatcher also implements common logic, such as `dispatch_many` to +dispatch a single event to multpiple keys in a single backend. This is useful +e.g a user is updated and you want to dispatch `USER_UPDATE` events to many +guilds without having to write a loop yourself. + +## Backend superclasses (litecord.pubsub.dispatcher) + +The backend superclasses define what methods backends must provide to be +fully functional within EventDispatcher. They define e.g what is the type +of the keys, and have some specific backend helper methods, such as +`_dispatch_states` to dispatch an event to a list of states without +worrying about errors or writing the loop. + +The other available superclass is `DispatchWithState` for backends that +require a list of subscribers to not repeat code. The only required method +to be implemented is `dispatch()` and you can see how that works out +on the backends that inherit from this class. From 3239a8c18a481ee0f93578eff1448706dd54c942 Mon Sep 17 00:00:00 2001 From: Luna Date: Wed, 20 Mar 2019 00:37:30 -0300 Subject: [PATCH 02/15] docs/pubsub.md: add practical call --- docs/pubsub.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/pubsub.md b/docs/pubsub.md index b8effc8..66dc0fe 100644 --- a/docs/pubsub.md +++ b/docs/pubsub.md @@ -55,3 +55,12 @@ The other available superclass is `DispatchWithState` for backends that require a list of subscribers to not repeat code. The only required method to be implemented is `dispatch()` and you can see how that works out on the backends that inherit from this class. + +## Sending an event, practical + +Call `app.dispatcher.dispatch(backend_string, key, event_type, event_payload)`. + +example: + - `dispatch('guild', guild_id, 'GUILD_UPDATE', guild)`, and other backends. + The rules on how each backend dispatches its events can be found on the + specific backend class. From d1c0a7aa3c5011d34547b7d78e1937281b396989 Mon Sep 17 00:00:00 2001 From: Luna Date: Wed, 20 Mar 2019 01:27:52 -0300 Subject: [PATCH 03/15] add docs/structure.md --- docs/structure.md | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 docs/structure.md diff --git a/docs/structure.md b/docs/structure.md new file mode 100644 index 0000000..f5e0fea --- /dev/null +++ b/docs/structure.md @@ -0,0 +1,42 @@ +# Project structure + +## `attachments` and `images` + +They're empty folders on purpose. Litecord will write files to them to hold +message attachments or avatars. + +## `manage` + +Contains the `manage.py` script's main function, plus all the commands. +A point of interest is the `manage/cmd/migration/scripts` folder, as they hold +all the SQL scripts required for migrations. + +## `litecord` + +The folder + `run.py` contain all of the backend's source code. The backend runs +Quart as its HTTP server, and a `websockets` server for the Gateway. + + - `litecord/blueprints` for everything HTTP related. + - `litecord/gateway` for main things related to the websocket or the Gateway. + - `litecord/embed` contains code related to embed sanitization, schemas, and + mediaproxy contact. + - `litecord/ratelimits` hold the ratelimit implementation copied from + discord.py plus a simple manager to hold the ratelimit objects. a point of + interest is `litecord/ratelimits/handler.py` that holds the main thing. + - `litecord/pubsub` is defined on `docs/pubsub.md`. + - `litecord/voice` holds the voice implementation, LVSP client, etc. + +There are other files around `litecord/`, e.g the snowflake library, presence/ +image/job managers, etc. + +## `static` + +Holds static files, such as a basic index page and the `invite_register.html` +page. + +## `tests` + +Tests are run with `pytest` and the asyncio plugin for proper testing. A point +of interest is `tests/conftest.py` as it contains test-specific configuration +for the app object. Adding a test is trivial, as pytest will match against any +file containing `test_` as a prefix. From 970b4b86928034d9f4acfe3ef89b5db9f34b56b2 Mon Sep 17 00:00:00 2001 From: Luna Date: Wed, 20 Mar 2019 01:40:23 -0300 Subject: [PATCH 04/15] litecord.schemas: add better docstring for validate() --- litecord/schemas.py | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/litecord/schemas.py b/litecord/schemas.py index 980a7ff..f28c1b0 100644 --- a/litecord/schemas.py +++ b/litecord/schemas.py @@ -49,7 +49,8 @@ EMOJO_MENTION = re.compile(r'<:(\.+):(\d+)>', re.A | re.M) ANIMOJI_MENTION = re.compile(r'', re.A | re.M) -def _in_enum(enum, value: int): +def _in_enum(enum, value) -> bool: + """Return if a given value is in the enum.""" try: enum(value) return True @@ -58,6 +59,7 @@ def _in_enum(enum, value: int): class LitecordValidator(Validator): + """Main validator class for Litecord, containing custom types.""" def _validate_type_username(self, value: str) -> bool: """Validate against the username regex.""" return bool(USERNAME_REGEX.match(value)) @@ -88,6 +90,10 @@ class LitecordValidator(Validator): return False def _validate_type_voice_region(self, value: str) -> bool: + # TODO: call voice manager for regions instead of hardcoding + + # I'm sure the context would be there at least in a basic level, so + # we can access the app. return value.lower() in ('brazil', 'us-east', 'us-west', 'us-south', 'russia') @@ -118,6 +124,7 @@ class LitecordValidator(Validator): except (TypeError, ValueError): return False + # nobody is allowed to use the INCOMING and OUTGOING rel types return val in (RelationshipType.FRIEND.value, RelationshipType.BLOCK.value) @@ -148,8 +155,18 @@ class LitecordValidator(Validator): def validate(reqjson: Union[Dict, List], schema: Dict, raise_err: bool = True) -> Dict: - """Validate a given document (user-input) and give - the correct document as a result. + """Validate the given user-given data against a schema, giving the + "correct" version of the document, with all defaults applied. + + Parameters + ---------- + reqjson: + The input data + schema: + The schema to validate reqjson against + raise_err: + If we should raise a BadRequest error when the validation + fails. Default is true. """ validator = LitecordValidator(schema) From 69723437a618ab5fb12eb4956bd5e203d7b79825 Mon Sep 17 00:00:00 2001 From: Luna Date: Wed, 20 Mar 2019 01:51:34 -0300 Subject: [PATCH 05/15] litecord.auth: add docstrings and typings --- litecord/auth.py | 38 ++++++++++++++++++++++++++++++-------- 1 file changed, 30 insertions(+), 8 deletions(-) diff --git a/litecord/auth.py b/litecord/auth.py index 715865f..b52e1a1 100644 --- a/litecord/auth.py +++ b/litecord/auth.py @@ -20,6 +20,7 @@ along with this program. If not, see . import base64 import binascii from random import randint +from typing import Tuple import bcrypt from asyncpg import UniqueViolationError @@ -35,7 +36,21 @@ from litecord.enums import UserFlags log = Logger(__name__) -async def raw_token_check(token, db=None): +async def raw_token_check(token: str, db=None) -> int: + """Check if a given token is valid. + + Returns + ------- + int + The User ID of the given token. + + Raises + ------ + Unauthorized + If token is not properly formatted, or if the user does not exist. + Forbidden + If token validation fails. + """ db = db or app.db # just try by fragments instead of @@ -80,7 +95,7 @@ async def raw_token_check(token, db=None): raise Forbidden('Invalid token') -async def token_check(): +async def token_check() -> int: """Check token information.""" # first, check if the request info already has a uid try: @@ -101,7 +116,7 @@ async def token_check(): return user_id -async def admin_check(): +async def admin_check() -> int: """Check if the user is an admin.""" user_id = await token_check() @@ -148,13 +163,19 @@ async def check_username_usage(username: str, db=None): async def create_user(username: str, email: str, password: str, - db=None, loop=None): - """Create a single user.""" + db=None, loop=None) -> Tuple[int, str]: + """Create a single user. + + Generates a distriminator and other information. You can fetch the user + data back with :meth:`Storage.get_user`. + """ db = db or app.db loop = loop or app.loop new_id = get_snowflake() + # TODO: unified discrim generation based off username, that also includes + # the check_username_usage() new_discrim = randint(1, 9999) new_discrim = '%04d' % new_discrim @@ -164,9 +185,10 @@ async def create_user(username: str, email: str, password: str, try: await db.execute(""" - INSERT INTO users (id, email, username, - discriminator, password_hash) - VALUES ($1, $2, $3, $4, $5) + INSERT INTO users + (id, email, username, discriminator, password_hash) + VALUES + ($1, $2, $3, $4, $5) """, new_id, email, username, new_discrim, pwd_hash) except UniqueViolationError: raise BadRequest('Email already used.') From f1b8f7b0a0b576a0b22eda071ea6556de4a7c067 Mon Sep 17 00:00:00 2001 From: Luna Date: Wed, 20 Mar 2019 01:56:40 -0300 Subject: [PATCH 06/15] litecord.jobs: add docstrings --- litecord/jobs.py | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/litecord/jobs.py b/litecord/jobs.py index 55b8f1d..b27099e 100644 --- a/litecord/jobs.py +++ b/litecord/jobs.py @@ -24,18 +24,27 @@ log = Logger(__name__) class JobManager: - """Manage background jobs""" + """Background job manager. + + Handles closing all existing jobs when going on a shutdown. This does not + use helpers such as asyncio.gather and asyncio.Task.all_tasks. It only uses + its own internal list of jobs. + """ def __init__(self, loop=None): self.loop = loop or asyncio.get_event_loop() self.jobs = [] async def _wrapper(self, coro): + """Wrapper coroutine for other coroutines. This adds a simple + try/except for general exceptions to be logged. + """ try: await coro except Exception: log.exception('Error while running job') def spawn(self, coro): + """Spawn a given future or coroutine in the background.""" task = self.loop.create_task( self._wrapper(coro) ) @@ -43,5 +52,10 @@ class JobManager: self.jobs.append(task) def close(self): + """Close the job manager, cancelling all existing jobs. + + It is the job's responsibility to handle the given CancelledError + and release any acquired resources. + """ for job in self.jobs: job.cancel() From d20626e7167e93642039d4c0251e94a54dd81847 Mon Sep 17 00:00:00 2001 From: Luna Date: Wed, 20 Mar 2019 02:04:36 -0300 Subject: [PATCH 07/15] litecord.presence: add docstrings / better function naming --- litecord/presence.py | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/litecord/presence.py b/litecord/presence.py index bf7bad9..2c2b6a5 100644 --- a/litecord/presence.py +++ b/litecord/presence.py @@ -68,6 +68,12 @@ def _best_presence(shards): async def _pres(storage, user_id: int, status_obj: dict) -> dict: + """Convert a given status into a presence, given the User ID and the + :class:`Storage` instance. + + This adds the required `presences` array, that isn't used, due to Litecord's + lack of Rich Presence. + """ ext = { 'user': await storage.get_user(user_id), 'activities': [], @@ -77,7 +83,11 @@ async def _pres(storage, user_id: int, status_obj: dict) -> dict: class PresenceManager: - """Presence related functions.""" + """Presence management. + + Has common functions to deal with fetching or updating presences, including + side-effects (events). + """ def __init__(self, app): self.storage = app.storage self.user_storage = app.user_storage @@ -159,7 +169,9 @@ class PresenceManager: 'activities': [game] if game else [] } - def _sane_session(session_id): + # given a session id, return if the session id actually connects to + # a given user, and if the state has not been dispatched via lazy guild. + def _session_check(session_id): state = self.state_manager.fetch_raw(session_id) uid = int(member['user']['id']) @@ -175,8 +187,7 @@ class PresenceManager: # gets a PRESENCE_UPDATE await self.dispatcher.dispatch_filter( 'guild', guild_id, - _sane_session, - + _session_check, 'PRESENCE_UPDATE', pres_update_payload ) From 55dbe4f5a27d238ebcb69ffe934e02fdd067220a Mon Sep 17 00:00:00 2001 From: Luna Date: Wed, 20 Mar 2019 02:05:35 -0300 Subject: [PATCH 08/15] snowflake: remove unused get_invite_code blueprints.invites uses its own gen_inv_code() function, that is reasonably faster than the original. --- litecord/snowflake.py | 8 -------- 1 file changed, 8 deletions(-) diff --git a/litecord/snowflake.py b/litecord/snowflake.py index d121f7a..05de350 100644 --- a/litecord/snowflake.py +++ b/litecord/snowflake.py @@ -41,14 +41,6 @@ WORKER_ID = 1 Snowflake = int -def get_invite_code() -> str: - """Get a random invite code.""" - random_stuff = hashlib.sha512(os.urandom(1024)).digest() - code = base64.urlsafe_b64encode(random_stuff).decode().replace('=', '5') \ - .replace('_', 'W').replace('-', 'm') - return code[:6] - - def _snowflake(timestamp: int) -> Snowflake: """Get a snowflake from a specific timestamp From 30dae2bf0e9595dc597fe4528cf398ea8c09119c Mon Sep 17 00:00:00 2001 From: Luna Date: Wed, 20 Mar 2019 02:15:55 -0300 Subject: [PATCH 09/15] snowflake: remove unused imports --- litecord/snowflake.py | 3 --- 1 file changed, 3 deletions(-) diff --git a/litecord/snowflake.py b/litecord/snowflake.py index 05de350..fd85995 100644 --- a/litecord/snowflake.py +++ b/litecord/snowflake.py @@ -24,10 +24,7 @@ snowflake.py - snowflake helper functions File brought in from litecord-reference(https://github.com/lnmds/litecord-reference) """ -import os import time -import base64 -import hashlib import datetime # encoded in ms From c07a1e28bd24f76dff51737de3bb0e122830795e Mon Sep 17 00:00:00 2001 From: Luna Date: Wed, 20 Mar 2019 03:42:11 -0300 Subject: [PATCH 10/15] system_messages: add docstring for send_sys_message --- litecord/system_messages.py | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/litecord/system_messages.py b/litecord/system_messages.py index 0f13b8e..a9e1cab 100644 --- a/litecord/system_messages.py +++ b/litecord/system_messages.py @@ -133,7 +133,26 @@ async def _handle_gdm_icon_edit(app, channel_id, author_id): async def send_sys_message(app, channel_id: int, m_type: MessageType, *args, **kwargs) -> int: - """Send a system message.""" + """Send a system message. + + The handler for a given message type MUST return an integer, that integer + being the message ID generated. This function takes care of feching the + message and dispatching the needed event. + + Parameters + ---------- + app + The app instance. + channel_id + The channel ID to send the system message to. + m_type + The system message's type. + + Returns + ------- + int + The message ID. + """ try: handler = { MessageType.CHANNEL_PINNED_MESSAGE: _handle_pin_msg, From b0f881f422d8a194113d39c1f972c94a01203d84 Mon Sep 17 00:00:00 2001 From: Luna Date: Wed, 20 Mar 2019 17:27:35 -0300 Subject: [PATCH 11/15] utils: docstring arrangements --- litecord/utils.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/litecord/utils.py b/litecord/utils.py index d19df06..db34b28 100644 --- a/litecord/utils.py +++ b/litecord/utils.py @@ -39,6 +39,7 @@ async def async_map(function, iterable: Iterable) -> list: async def task_wrapper(name: str, coro): + """Wrap a given coroutine in a task.""" try: await coro except asyncio.CancelledError: @@ -136,7 +137,10 @@ def mmh3(inp_str: str, seed: int = 0): class LitecordJSONEncoder(JSONEncoder): + """Custom JSON encoder for Litecord.""" def default(self, value: Any): + """By default, this will try to get the to_json attribute of a given + value being JSON encoded.""" try: return value.to_json except AttributeError: @@ -144,8 +148,7 @@ class LitecordJSONEncoder(JSONEncoder): async def pg_set_json(con): - """Set JSON and JSONB codecs for an - asyncpg connection.""" + """Set JSON and JSONB codecs for an asyncpg connection.""" await con.set_type_codec( 'json', encoder=lambda v: json.dumps(v, cls=LitecordJSONEncoder), From 58834f88af3ed683a11fc16cc94dd00089e2dc2d Mon Sep 17 00:00:00 2001 From: Luna Date: Wed, 20 Mar 2019 17:34:58 -0300 Subject: [PATCH 12/15] permissions: add docstrings --- litecord/permissions.py | 39 ++++++++++++++++++++++++++++++++------- 1 file changed, 32 insertions(+), 7 deletions(-) diff --git a/litecord/permissions.py b/litecord/permissions.py index 4b062e8..d0ebb75 100644 --- a/litecord/permissions.py +++ b/litecord/permissions.py @@ -64,6 +64,14 @@ class _RawPermsBits(ctypes.LittleEndianStructure): class Permissions(ctypes.Union): + """Main permissions class. Holds helper functions to convert between + the bitfield and an integer, etc. + + Parameters + ---------- + val + The permissions value as an integer. + """ _fields_ = [ ('bits', _RawPermsBits), ('binary', ctypes.c_uint64), @@ -78,9 +86,6 @@ class Permissions(ctypes.Union): def __int__(self): return self.binary - def numby(self): - return self.binary - ALL_PERMISSIONS = Permissions(0b01111111111101111111110111111111) @@ -152,6 +157,7 @@ async def base_permissions(member_id, guild_id, storage=None) -> Permissions: def overwrite_mix(perms: Permissions, overwrite: dict) -> Permissions: + """Mix a single permission with a single overwrite.""" # we make a copy of the binary representation # so we don't modify the old perms in-place # which could be an unwanted side-effect @@ -168,6 +174,24 @@ def overwrite_mix(perms: Permissions, overwrite: dict) -> Permissions: def overwrite_find_mix(perms: Permissions, overwrites: dict, target_id: int) -> Permissions: + """Mix a given permission with a given overwrite. + + Returns the given permission if an overwrite is not found. + + Parameters + ---------- + perms + The permissions for the given target. + overwrites + The overwrites for the given actor (mostly channel). + target_id + The target's ID in the overwrites dict. + + Returns + ------- + Permissions + The mixed permissions object. + """ overwrite = overwrites.get(target_id) if overwrite: @@ -251,8 +275,9 @@ async def compute_overwrites(base_perms: Permissions, return perms -async def get_permissions(member_id, channel_id, *, storage=None): - """Get all the permissions for a user in a channel.""" +async def get_permissions(member_id: int, channel_id, + *, storage=None) -> Permissions: + """Get the permissions for a user in a channel.""" if not storage: storage = app.storage @@ -264,5 +289,5 @@ async def get_permissions(member_id, channel_id, *, storage=None): base_perms = await base_permissions(member_id, guild_id, storage) - return await compute_overwrites(base_perms, member_id, - channel_id, guild_id, storage) + return await compute_overwrites( + base_perms, member_id, channel_id, guild_id, storage) From 3b85556d89ad61be007b2b96149e5cf7bf7969a4 Mon Sep 17 00:00:00 2001 From: Luna Date: Wed, 20 Mar 2019 17:54:24 -0300 Subject: [PATCH 13/15] gateway: docstrings! also subjective readability improvements --- litecord/gateway/gateway.py | 12 +++++---- litecord/gateway/websocket.py | 50 +++++++++++++++++++++-------------- 2 files changed, 37 insertions(+), 25 deletions(-) diff --git a/litecord/gateway/gateway.py b/litecord/gateway/gateway.py index d6ec58a..f435d16 100644 --- a/litecord/gateway/gateway.py +++ b/litecord/gateway/gateway.py @@ -22,9 +22,8 @@ from litecord.gateway.websocket import GatewayWebsocket async def websocket_handler(app, ws, url): - """Main websocket handler, checks query arguments - when connecting to the gateway and spawns a - GatewayWebsocket instance for the connection.""" + """Main websocket handler, checks query arguments when connecting to + the gateway and spawns a GatewayWebsocket instance for the connection.""" args = urllib.parse.parse_qs( urllib.parse.urlparse(url).query ) @@ -54,6 +53,9 @@ async def websocket_handler(app, ws, url): if gw_compress and gw_compress not in ('zlib-stream',): return await ws.close(1000, 'Invalid gateway compress') - gws = GatewayWebsocket(ws, app, v=gw_version, - encoding=gw_encoding, compress=gw_compress) + gws = GatewayWebsocket( + ws, app, v=gw_version, encoding=gw_encoding, compress=gw_compress) + + # this can be run with a single await since this whole coroutine + # is already running in the background. await gws.run() diff --git a/litecord/gateway/websocket.py b/litecord/gateway/websocket.py index 4cc22aa..436001a 100644 --- a/litecord/gateway/websocket.py +++ b/litecord/gateway/websocket.py @@ -65,30 +65,34 @@ WebsocketObjects = collections.namedtuple( def encode_json(payload) -> str: + """Encode a given payload to JSON.""" return json.dumps(payload, separators=(',', ':'), cls=LitecordJSONEncoder) def decode_json(data: str): + """Decode from JSON.""" return json.loads(data) def encode_etf(payload) -> str: - # The thing with encoding ETF is that with json we have LitecordJSONEncoder - # which takes care of converting e.g datetime objects to their ISO - # representation. + """Encode a payload to ETF (External Term Format). - # so, to keep things working, i'll to a json pass on the payload, then send - # the decoded payload back to earl. + This gives a JSON pass on the given payload (via calling encode_json and + then decode_json) because we may want to encode objects that can only be + encoded by LitecordJSONEncoder. + Earl-ETF does not give the same interface for extensibility, hence why we + do the pass. + """ sanitized = encode_json(payload) sanitized = decode_json(sanitized) return earl.pack(sanitized) def _etf_decode_dict(data): - # NOTE: this is a very slow implementation to - # decode the dictionary. + """Decode a given dictionary.""" + # NOTE: this is very slow. if isinstance(data, bytes): return data.decode() @@ -109,6 +113,7 @@ def _etf_decode_dict(data): return result def decode_etf(data: bytes): + """Decode data in ETF to any.""" res = earl.unpack(data) if isinstance(res, bytes): @@ -269,7 +274,7 @@ class GatewayWebsocket: task_wrapper('hb wait', self._hb_wait(interval)) ) - async def send_hello(self): + async def _send_hello(self): """Send the OP 10 Hello packet over the websocket.""" # random heartbeat intervals interval = randint(40, 46) * 1000 @@ -367,7 +372,7 @@ class GatewayWebsocket: 'friend_suggestion_count': 0, - + # those are unused default values. 'connected_accounts': [], 'experiments': [], 'guild_experiments': [], @@ -387,7 +392,7 @@ class GatewayWebsocket: uready = await self._user_ready() private_channels = ( - await self.user_storage.get_dms(user_id) + + await self.user_storage.get_dms(user_id) + await self.user_storage.get_gdms(user_id) ) @@ -406,6 +411,8 @@ class GatewayWebsocket: self.ext.loop.create_task(self._guild_dispatch(guilds)) async def _check_shards(self, shard, user_id): + """Check if the given `shard` value in IDENTIFY has good enough values. + """ current_shard, shard_count = shard guilds = await self.ext.db.fetchval(""" @@ -612,10 +619,6 @@ class GatewayWebsocket: # setting new presence to state await self.update_status(presence) - def voice_key(self, channel_id: int, guild_id: int): - """Voice state key.""" - return (self.state.user_id, self.state.session_id) - async def _vsu_get_prop(self, state, data): """Get voice state properties from data, fallbacking to user settings.""" @@ -837,6 +840,11 @@ class GatewayWebsocket: await self._req_guild_members(gid, [], query, limit) async def _guild_sync(self, guild_id: int): + """Synchronize a guild. + + Fetches the members and presences of a guild and dispatches a + GUILD_SYNC event with that info. + """ members = await self.storage.get_member_data(guild_id) member_ids = [int(m['user']['id']) for m in members] @@ -979,7 +987,7 @@ class GatewayWebsocket: self.state.session_id, ranges ) - async def process_message(self, payload): + async def _process_message(self, payload): """Process a single message coming in from the client.""" try: op_code = payload['op'] @@ -998,7 +1006,7 @@ class GatewayWebsocket: if self._check_ratelimit('messages', self.state.session_id): raise WebsocketClose(4008, 'You are being ratelimited.') - async def listen_messages(self): + async def _listen_messages(self): """Listen for messages coming in from the websocket.""" # close anyone trying to login while the @@ -1018,9 +1026,11 @@ class GatewayWebsocket: await self._msg_ratelimit() payload = self.decoder(message) - await self.process_message(payload) + await self._process_message(payload) def _cleanup(self): + """Cleanup any leftover tasks, and remove the connection from the + state manager.""" for task in self.wsp.tasks.values(): task.cancel() @@ -1058,11 +1068,11 @@ class GatewayWebsocket: ) async def run(self): - """Wrap listen_messages inside + """Wrap :meth:`listen_messages` inside a try/except block for WebsocketClose handling.""" try: - await self.send_hello() - await self.listen_messages() + await self._send_hello() + await self._listen_messages() except websockets.exceptions.ConnectionClosed as err: log.warning('conn close, state={}, err={}', self.state, err) except WebsocketClose as err: From dc5aeda06bf5b1426cc69a8038b985fa2d3c11c1 Mon Sep 17 00:00:00 2001 From: Luna Date: Fri, 22 Mar 2019 19:04:36 -0300 Subject: [PATCH 14/15] README: add note on docs/operating.md --- README.md | 5 ++++- docs/operating.md | 15 +++++++++++++++ 2 files changed, 19 insertions(+), 1 deletion(-) create mode 100644 docs/operating.md diff --git a/README.md b/README.md index 4ee38a0..fb1e849 100644 --- a/README.md +++ b/README.md @@ -66,7 +66,7 @@ or third party libraries (such as [Eris](https://github.com/abalabahaha/eris)). Requirements: - **Python 3.7+** -- PostgreSQL (tested using 9.6+) +- PostgreSQL (tested using 9.6+), SQL knowledge is recommended. - gifsicle for GIF emoji and avatar handling - [pipenv] @@ -133,6 +133,9 @@ You can check if your instance is running by performing an HTTP `GET` request on the `/api/v6/gateway` endpoint. For basic websocket testing, a tool such as [ws](https://github.com/hashrocket/ws) can be used. +After checking that it actually works, `docs/operating.md` continues on common +operations for a Litecord instance. + ## Updating Update the code and run any new database migrations: diff --git a/docs/operating.md b/docs/operating.md new file mode 100644 index 0000000..b859cfe --- /dev/null +++ b/docs/operating.md @@ -0,0 +1,15 @@ +# Operating a Litecord instance + +`./manage.py` contains common admin tasks that you may want to do to the +instance, e.g make someone an admin, or migrate the database, etc. + +Note, however, that many commands (example the ones related to user deletion) +may be moved to the Admin API without proper notice. There is no frontend yet +for the Admin API. + +The possible actions on `./manage.py` can be accessed via `./manage.py -h`, or +`pipenv run ./manage.py -h` if you're on pipenv. + +## `./manage.py generate_token`? + +You can generate a user token but only if that user is a bot. From 5df68896d0af81a5d35bcf75e5bf1600364b4272 Mon Sep 17 00:00:00 2001 From: Luna Date: Fri, 22 Mar 2019 19:13:40 -0300 Subject: [PATCH 15/15] docs/operating.md: add 'Instance Invites' and 'Making someone Staff' --- docs/operating.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/operating.md b/docs/operating.md index b859cfe..b30fe3f 100644 --- a/docs/operating.md +++ b/docs/operating.md @@ -13,3 +13,22 @@ The possible actions on `./manage.py` can be accessed via `./manage.py -h`, or ## `./manage.py generate_token`? You can generate a user token but only if that user is a bot. + +## Instance invites + +If your instance has registrations disabled you can still get users to the +instance via instance invites. This is something only Litecord does, using a +separate API endpoint. + +Use `./manage.py makeinv` to generate an instance invite, give it out to users, +point them towards `https:///invite_register.html`. Things +should be straightforward from there. + +## Making someone Staff + +**CAUTION:** Making someone staff, other than giving the Staff badge on their +user flags, also gives complete access over the Admin API. Only make staff the +people you (the instance OP) can trust. + +Use the `./manage.py make_staff` management task to make someone staff. There is +no way to remove someone's staff with a `./manage.py` command *yet.*