Merge branch 'internal-docs' into 'master'

Documentation Improvements

Closes #42

See merge request litecord/litecord!28

README.md

@@ -66,7 +66,7 @@ or third party libraries (such as [Eris](https://github.com/abalabahaha/eris)).
 Requirements:
 
 - **Python 3.7+**
-- PostgreSQL (tested using 9.6+)
+- PostgreSQL (tested using 9.6+), SQL knowledge is recommended.
 - gifsicle for GIF emoji and avatar handling
 - [pipenv]
 
@@ -133,6 +133,9 @@ You can check if your instance is running by performing an HTTP `GET` request on
 the `/api/v6/gateway` endpoint. For basic websocket testing, a tool such as
 [ws](https://github.com/hashrocket/ws) can be used.
 
+After checking that it actually works, `docs/operating.md` continues on common
+operations for a Litecord instance.
+
 ## Updating
 
 Update the code and run any new database migrations:

docs/README.md

@@ -1,3 +1,5 @@
 # Internal documentation
 
-The Litecord Voice Server Protocol (LVSP) is documented here.
+ - `admin_api.md` for Litecord's Admin API.
+ - `lvsp.md` for the Litecord Voice Server Protocol.
+ - `pubsub.md` for how Pub/Sub works in Litecord.

docs/operating.md

@@ -0,0 +1,34 @@
+# Operating a Litecord instance
+
+`./manage.py` contains common admin tasks that you may want to do to the
+instance, e.g make someone an admin, or migrate the database, etc.
+
+Note, however, that many commands (example the ones related to user deletion)
+may be moved to the Admin API without proper notice. There is no frontend yet
+for the Admin API.
+
+The possible actions on `./manage.py` can be accessed via `./manage.py -h`, or
+`pipenv run ./manage.py -h` if you're on pipenv.
+
+## `./manage.py generate_token`?
+
+You can generate a user token but only if that user is a bot.
+
+## Instance invites
+
+If your instance has registrations disabled you can still get users to the
+instance via instance invites. This is something only Litecord does, using a
+separate API endpoint.
+
+Use `./manage.py makeinv` to generate an instance invite, give it out to users,
+point them towards `https://<your instance url>/invite_register.html`. Things
+should be straightforward from there.
+
+## Making someone Staff
+
+**CAUTION:** Making someone staff, other than giving the Staff badge on their
+user flags, also gives complete access over the Admin API. Only make staff the
+people you (the instance OP) can trust.
+
+Use the `./manage.py make_staff` management task to make someone staff. There is
+no way to remove someone's staff with a `./manage.py` command *yet.*

docs/pubsub.md

@@ -0,0 +1,66 @@
+# Pub/Sub (Publish/Subscribe)
+
+Please look over wikipedia or other sources to understand what it is.
+This only documents how an events are generated and dispatched to clients.
+
+## Event definition
+
+Events are composed of two things:
+ - Event type
+ - Event payload
+
+More information on how events are structured are in the Discord Gateway
+API documentation.
+
+## `StateManager` (litecord.gateway.state\_manager)
+
+StateManager stores all available instances of `GatewayState` that identified.
+Specific information is over the class' docstring, but we at least define
+what it does here for the next class:
+
+## `EventDispatcher` (litecord.dispatcher)
+
+EventDispatcher declares the main interface between clients and the side-effects
+(events) from topics they can subscribe to.
+
+The topic / channel in EventDispatcher can be a User, or a Guild, or a Channel,
+etc. Users subscribe to the channels they have access to manually, and get
+unsubscribed when they e.g leave the guild the channel is on, etc.
+
+Channels are identified by their backend and a given key uniquely identifying
+the channel. The backend can be `guild`, `member`, `channel`, `user`,
+`friend`, and `lazy_guild`. Backends *can* store the list of subscribers, but
+that is not required.
+
+Each backend has specific logic around dispatching a single event towards
+all the subscribers of the given key. For example, the `guild` backend only
+dispatches events to shards that are properly subscribed to the guild,
+instead of all shards. The `user` backend just dispatches the event to
+all available shards in `StateManager`, etc.
+
+EventDispatcher also implements common logic, such as `dispatch_many` to
+dispatch a single event to multpiple keys in a single backend. This is useful
+e.g a user is updated and you want to dispatch `USER_UPDATE` events to many
+guilds without having to write a loop yourself.
+
+## Backend superclasses (litecord.pubsub.dispatcher)
+
+The backend superclasses define what methods backends must provide to be
+fully functional within EventDispatcher. They define e.g what is the type
+of the keys, and have some specific backend helper methods, such as
+`_dispatch_states` to dispatch an event to a list of states without
+worrying about errors or writing the loop.
+
+The other available superclass is `DispatchWithState` for backends that
+require a list of subscribers to not repeat code. The only required method
+to be implemented is `dispatch()` and you can see how that works out
+on the backends that inherit from this class.
+
+## Sending an event, practical
+
+Call `app.dispatcher.dispatch(backend_string, key, event_type, event_payload)`.
+
+example:
+ - `dispatch('guild', guild_id, 'GUILD_UPDATE', guild)`, and other backends.
+    The rules on how each backend dispatches its events can be found on the
+    specific backend class.

docs/structure.md

@@ -0,0 +1,42 @@
+# Project structure
+
+## `attachments` and `images`
+
+They're empty folders on purpose. Litecord will write files to them to hold
+message attachments or avatars.
+
+## `manage`
+
+Contains the `manage.py` script's main function, plus all the commands.
+A point of interest is the `manage/cmd/migration/scripts` folder, as they hold
+all the SQL scripts required for migrations.
+
+## `litecord`
+
+The folder + `run.py` contain all of the backend's source code. The backend runs
+Quart as its HTTP server, and a `websockets` server for the Gateway.
+
+ - `litecord/blueprints` for everything HTTP related. 
+ - `litecord/gateway` for main things related to the websocket or the Gateway.
+ - `litecord/embed` contains code related to embed sanitization, schemas, and
+    mediaproxy contact.
+ - `litecord/ratelimits` hold the ratelimit implementation copied from
+    discord.py plus a simple manager to hold the ratelimit objects. a point of
+    interest is `litecord/ratelimits/handler.py` that holds the main thing.
+ - `litecord/pubsub` is defined on `docs/pubsub.md`.
+ - `litecord/voice` holds the voice implementation, LVSP client, etc.
+
+There are other files around `litecord/`, e.g the snowflake library, presence/
+image/job managers, etc.
+
+## `static`
+
+Holds static files, such as a basic index page and the `invite_register.html`
+page.
+
+## `tests`
+
+Tests are run with `pytest` and the asyncio plugin for proper testing. A point
+of interest is `tests/conftest.py` as it contains test-specific configuration
+for the app object. Adding a test is trivial, as pytest will match against any
+file containing `test_` as a prefix.

litecord/auth.py

@@ -20,6 +20,7 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
 import base64
 import binascii
 from random import randint
+from typing import Tuple
 
 import bcrypt
 from asyncpg import UniqueViolationError
@@ -35,7 +36,21 @@ from litecord.enums import UserFlags
 log = Logger(__name__)
 
 
-async def raw_token_check(token, db=None):
+async def raw_token_check(token: str, db=None) -> int:
+    """Check if a given token is valid.
+
+    Returns
+    -------
+    int
+        The User ID of the given token.
+
+    Raises
+    ------
+    Unauthorized
+        If token is not properly formatted, or if the user does not exist.
+    Forbidden
+        If token validation fails.
+    """
     db = db or app.db
 
     # just try by fragments instead of
@@ -80,7 +95,7 @@ async def raw_token_check(token, db=None):
         raise Forbidden('Invalid token')
 
 
-async def token_check():
+async def token_check() -> int:
     """Check token information."""
     # first, check if the request info already has a uid
     try:
@@ -101,7 +116,7 @@ async def token_check():
     return user_id
 
 
-async def admin_check():
+async def admin_check() -> int:
     """Check if the user is an admin."""
     user_id = await token_check()
 
@@ -148,13 +163,19 @@ async def check_username_usage(username: str, db=None):
 
 
 async def create_user(username: str, email: str, password: str,
-                      db=None, loop=None):
-    """Create a single user."""
+                      db=None, loop=None) -> Tuple[int, str]:
+    """Create a single user.
+
+    Generates a distriminator and other information. You can fetch the user
+    data back with :meth:`Storage.get_user`.
+    """
     db = db or app.db
     loop = loop or app.loop
 
     new_id = get_snowflake()
 
+    # TODO: unified discrim generation based off username, that also includes
+    # the check_username_usage()
     new_discrim = randint(1, 9999)
     new_discrim = '%04d' % new_discrim
 
@@ -164,9 +185,10 @@ async def create_user(username: str, email: str, password: str,
 
     try:
         await db.execute("""
-        INSERT INTO users (id, email, username,
-                        discriminator, password_hash)
-        VALUES ($1, $2, $3, $4, $5)
+        INSERT INTO users
+            (id, email, username, discriminator, password_hash)
+        VALUES
+            ($1, $2, $3, $4, $5)
         """, new_id, email, username, new_discrim, pwd_hash)
     except UniqueViolationError:
         raise BadRequest('Email already used.')

litecord/gateway/gateway.py

@@ -22,9 +22,8 @@ from litecord.gateway.websocket import GatewayWebsocket
 
 
 async def websocket_handler(app, ws, url):
-    """Main websocket handler, checks query arguments
-    when connecting to the gateway and spawns a
-    GatewayWebsocket instance for the connection."""
+    """Main websocket handler, checks query arguments when connecting to
+    the gateway and spawns a GatewayWebsocket instance for the connection."""
     args = urllib.parse.parse_qs(
         urllib.parse.urlparse(url).query
     )
@@ -54,6 +53,9 @@ async def websocket_handler(app, ws, url):
     if gw_compress and gw_compress not in ('zlib-stream',):
         return await ws.close(1000, 'Invalid gateway compress')
 
-    gws = GatewayWebsocket(ws, app, v=gw_version,
-                           encoding=gw_encoding, compress=gw_compress)
+    gws = GatewayWebsocket(
+        ws, app, v=gw_version, encoding=gw_encoding, compress=gw_compress)
+
+    # this can be run with a single await since this whole coroutine
+    # is already running in the background.
     await gws.run()

litecord/gateway/websocket.py

@@ -65,30 +65,34 @@ WebsocketObjects = collections.namedtuple(
 
 
 def encode_json(payload) -> str:
+    """Encode a given payload to JSON."""
     return json.dumps(payload, separators=(',', ':'),
                       cls=LitecordJSONEncoder)
 
 
 def decode_json(data: str):
+    """Decode from JSON."""
     return json.loads(data)
 
 
 def encode_etf(payload) -> str:
-    # The thing with encoding ETF is that with json we have LitecordJSONEncoder
-    # which takes care of converting e.g datetime objects to their ISO
-    # representation.
+    """Encode a payload to ETF (External Term Format).
 
-    # so, to keep things working, i'll to a json pass on the payload, then send
-    # the decoded payload back to earl.
+    This gives a JSON pass on the given payload (via calling encode_json and
+    then decode_json) because we may want to encode objects that can only be
+    encoded by LitecordJSONEncoder.
 
+    Earl-ETF does not give the same interface for extensibility, hence why we
+    do the pass.
+    """
     sanitized = encode_json(payload)
     sanitized = decode_json(sanitized)
     return earl.pack(sanitized)
 
 
 def _etf_decode_dict(data):
-    # NOTE: this is a very slow implementation to
-    # decode the dictionary.
+    """Decode a given dictionary."""
+    # NOTE: this is very slow.
 
     if isinstance(data, bytes):
         return data.decode()
@@ -109,6 +113,7 @@ def _etf_decode_dict(data):
     return result
 
 def decode_etf(data: bytes):
+    """Decode data in ETF to any."""
     res = earl.unpack(data)
 
     if isinstance(res, bytes):
@@ -269,7 +274,7 @@ class GatewayWebsocket:
             task_wrapper('hb wait', self._hb_wait(interval))
         )
 
-    async def send_hello(self):
+    async def _send_hello(self):
         """Send the OP 10 Hello packet over the websocket."""
         # random heartbeat intervals
         interval = randint(40, 46) * 1000
@@ -367,7 +372,7 @@ class GatewayWebsocket:
 
             'friend_suggestion_count': 0,
 
-
+            # those are unused default values.
             'connected_accounts': [],
             'experiments': [],
             'guild_experiments': [],
@@ -406,6 +411,8 @@ class GatewayWebsocket:
         self.ext.loop.create_task(self._guild_dispatch(guilds))
 
     async def _check_shards(self, shard, user_id):
+        """Check if the given `shard` value in IDENTIFY has good enough values.
+        """
         current_shard, shard_count = shard
 
         guilds = await self.ext.db.fetchval("""
@@ -612,10 +619,6 @@ class GatewayWebsocket:
         # setting new presence to state
         await self.update_status(presence)
 
-    def voice_key(self, channel_id: int, guild_id: int):
-        """Voice state key."""
-        return (self.state.user_id, self.state.session_id)
-
     async def _vsu_get_prop(self, state, data):
         """Get voice state properties from data, fallbacking to
         user settings."""
@@ -837,6 +840,11 @@ class GatewayWebsocket:
             await self._req_guild_members(gid, [], query, limit)
 
     async def _guild_sync(self, guild_id: int):
+        """Synchronize a guild.
+
+        Fetches the members and presences of a guild and dispatches a
+        GUILD_SYNC event with that info.
+        """
         members = await self.storage.get_member_data(guild_id)
         member_ids = [int(m['user']['id']) for m in members]
 
@@ -979,7 +987,7 @@ class GatewayWebsocket:
                 self.state.session_id, ranges
             )
 
-    async def process_message(self, payload):
+    async def _process_message(self, payload):
         """Process a single message coming in from the client."""
         try:
             op_code = payload['op']
@@ -998,7 +1006,7 @@ class GatewayWebsocket:
         if self._check_ratelimit('messages', self.state.session_id):
             raise WebsocketClose(4008, 'You are being ratelimited.')
 
-    async def listen_messages(self):
+    async def _listen_messages(self):
         """Listen for messages coming in from the websocket."""
 
         # close anyone trying to login while the
@@ -1018,9 +1026,11 @@ class GatewayWebsocket:
                 await self._msg_ratelimit()
 
             payload = self.decoder(message)
-            await self.process_message(payload)
+            await self._process_message(payload)
 
     def _cleanup(self):
+        """Cleanup any leftover tasks, and remove the connection from the
+        state manager."""
         for task in self.wsp.tasks.values():
             task.cancel()
 
@@ -1058,11 +1068,11 @@ class GatewayWebsocket:
             )
 
     async def run(self):
-        """Wrap listen_messages inside
+        """Wrap :meth:`listen_messages` inside
         a try/except block for WebsocketClose handling."""
         try:
-            await self.send_hello()
-            await self.listen_messages()
+            await self._send_hello()
+            await self._listen_messages()
         except websockets.exceptions.ConnectionClosed as err:
             log.warning('conn close, state={}, err={}', self.state, err)
         except WebsocketClose as err:

litecord/jobs.py

@@ -24,18 +24,27 @@ log = Logger(__name__)
 
 
 class JobManager:
-    """Manage background jobs"""
+    """Background job manager.
+
+    Handles closing all existing jobs when going on a shutdown. This does not
+    use helpers such as asyncio.gather and asyncio.Task.all_tasks. It only uses
+    its own internal list of jobs.
+    """
     def __init__(self, loop=None):
         self.loop = loop or asyncio.get_event_loop()
         self.jobs = []
 
     async def _wrapper(self, coro):
+        """Wrapper coroutine for other coroutines. This adds a simple
+        try/except for general exceptions to be logged.
+        """
         try:
             await coro
         except Exception:
             log.exception('Error while running job')
 
     def spawn(self, coro):
+        """Spawn a given future or coroutine in the background."""
         task = self.loop.create_task(
             self._wrapper(coro)
         )
@@ -43,5 +52,10 @@ class JobManager:
         self.jobs.append(task)
 
     def close(self):
+        """Close the job manager, cancelling all existing jobs.
+
+        It is the job's responsibility to handle the given CancelledError
+        and release any acquired resources.
+        """
         for job in self.jobs:
             job.cancel()

litecord/permissions.py

@@ -64,6 +64,14 @@ class _RawPermsBits(ctypes.LittleEndianStructure):
 
 
 class Permissions(ctypes.Union):
+    """Main permissions class. Holds helper functions to convert between
+    the bitfield and an integer, etc.
+
+    Parameters
+    ----------
+    val
+        The permissions value as an integer.
+    """
     _fields_ = [
         ('bits', _RawPermsBits),
         ('binary', ctypes.c_uint64),
@@ -78,9 +86,6 @@ class Permissions(ctypes.Union):
     def __int__(self):
         return self.binary
 
-    def numby(self):
-        return self.binary
-
 
 ALL_PERMISSIONS = Permissions(0b01111111111101111111110111111111)
 
@@ -152,6 +157,7 @@ async def base_permissions(member_id, guild_id, storage=None) -> Permissions:
 
 
 def overwrite_mix(perms: Permissions, overwrite: dict) -> Permissions:
+    """Mix a single permission with a single overwrite."""
     # we make a copy of the binary representation
     # so we don't modify the old perms in-place
     # which could be an unwanted side-effect
@@ -168,6 +174,24 @@ def overwrite_mix(perms: Permissions, overwrite: dict) -> Permissions:
 
 def overwrite_find_mix(perms: Permissions, overwrites: dict,
                        target_id: int) -> Permissions:
+    """Mix a given permission with a given overwrite.
+
+    Returns the given permission if an overwrite is not found.
+
+    Parameters
+    ----------
+    perms
+        The permissions for the given target.
+    overwrites
+        The overwrites for the given actor (mostly channel).
+    target_id
+        The target's ID in the overwrites dict.
+
+    Returns
+    -------
+    Permissions
+        The mixed permissions object.
+    """
     overwrite = overwrites.get(target_id)
 
     if overwrite:
@@ -251,8 +275,9 @@ async def compute_overwrites(base_perms: Permissions,
     return perms
 
 
-async def get_permissions(member_id, channel_id, *, storage=None):
-    """Get all the permissions for a user in a channel."""
+async def get_permissions(member_id: int, channel_id,
+                          *, storage=None) -> Permissions:
+    """Get the permissions for a user in a channel."""
     if not storage:
         storage = app.storage
 
@@ -264,5 +289,5 @@ async def get_permissions(member_id, channel_id, *, storage=None):
 
     base_perms = await base_permissions(member_id, guild_id, storage)
 
-    return await compute_overwrites(base_perms, member_id,
-                                    channel_id, guild_id, storage)
+    return await compute_overwrites(
+        base_perms, member_id, channel_id, guild_id, storage)

litecord/presence.py

@@ -68,6 +68,12 @@ def _best_presence(shards):
 
 
 async def _pres(storage, user_id: int, status_obj: dict) -> dict:
+    """Convert a given status into a presence, given the User ID and the
+    :class:`Storage` instance.
+
+    This adds the required `presences` array, that isn't used, due to Litecord's
+    lack of Rich Presence.
+    """
     ext = {
         'user': await storage.get_user(user_id),
         'activities': [],
@@ -77,7 +83,11 @@ async def _pres(storage, user_id: int, status_obj: dict) -> dict:
 
 
 class PresenceManager:
-    """Presence related functions."""
+    """Presence management.
+
+    Has common functions to deal with fetching or updating presences, including
+    side-effects (events).
+    """
     def __init__(self, app):
         self.storage = app.storage
         self.user_storage = app.user_storage
@@ -159,7 +169,9 @@ class PresenceManager:
             'activities': [game] if game else []
         }
 
-        def _sane_session(session_id):
+        # given a session id, return if the session id actually connects to
+        # a given user, and if the state has not been dispatched via lazy guild.
+        def _session_check(session_id):
             state = self.state_manager.fetch_raw(session_id)
             uid = int(member['user']['id'])
 
@@ -175,8 +187,7 @@ class PresenceManager:
         # gets a PRESENCE_UPDATE
         await self.dispatcher.dispatch_filter(
             'guild', guild_id,
-            _sane_session,
-
+            _session_check,
             'PRESENCE_UPDATE', pres_update_payload
         )
 

litecord/schemas.py

@@ -49,7 +49,8 @@ EMOJO_MENTION = re.compile(r'<:(\.+):(\d+)>', re.A | re.M)
 ANIMOJI_MENTION = re.compile(r'<a:(\.+):(\d+)>', re.A | re.M)
 
 
-def _in_enum(enum, value: int):
+def _in_enum(enum, value) -> bool:
+    """Return if a given value is in the enum."""
     try:
         enum(value)
         return True
@@ -58,6 +59,7 @@ def _in_enum(enum, value: int):
 
 
 class LitecordValidator(Validator):
+    """Main validator class for Litecord, containing custom types."""
     def _validate_type_username(self, value: str) -> bool:
         """Validate against the username regex."""
         return bool(USERNAME_REGEX.match(value))
@@ -88,6 +90,10 @@ class LitecordValidator(Validator):
             return False
 
     def _validate_type_voice_region(self, value: str) -> bool:
+        # TODO: call voice manager for regions instead of hardcoding
+
+        # I'm sure the context would be there at least in a basic level, so
+        # we can access the app.
         return value.lower() in ('brazil', 'us-east', 'us-west',
                                  'us-south', 'russia')
 
@@ -118,6 +124,7 @@ class LitecordValidator(Validator):
         except (TypeError, ValueError):
             return False
 
+        # nobody is allowed to use the INCOMING and OUTGOING rel types
         return val in (RelationshipType.FRIEND.value,
                        RelationshipType.BLOCK.value)
 
@@ -148,8 +155,18 @@ class LitecordValidator(Validator):
 
 def validate(reqjson: Union[Dict, List], schema: Dict,
              raise_err: bool = True) -> Dict:
-    """Validate a given document (user-input) and give
-    the correct document as a result.
+    """Validate the given user-given data against a schema, giving the
+    "correct" version of the document, with all defaults applied.
+
+    Parameters
+    ----------
+    reqjson:
+        The input data
+    schema:
+        The schema to validate reqjson against
+    raise_err:
+        If we should raise a BadRequest error when the validation
+        fails. Default is true.
     """
     validator = LitecordValidator(schema)
 

litecord/snowflake.py

@@ -24,10 +24,7 @@ snowflake.py - snowflake helper functions
     File brought in from
         litecord-reference(https://github.com/lnmds/litecord-reference)
 """
-import os
 import time
-import base64
-import hashlib
 import datetime
 
 # encoded in ms
@@ -41,14 +38,6 @@ WORKER_ID = 1
 Snowflake = int
 
 
-def get_invite_code() -> str:
-    """Get a random invite code."""
-    random_stuff = hashlib.sha512(os.urandom(1024)).digest()
-    code = base64.urlsafe_b64encode(random_stuff).decode().replace('=', '5') \
-        .replace('_', 'W').replace('-', 'm')
-    return code[:6]
-
-
 def _snowflake(timestamp: int) -> Snowflake:
     """Get a snowflake from a specific timestamp
 

litecord/system_messages.py

@@ -133,7 +133,26 @@ async def _handle_gdm_icon_edit(app, channel_id, author_id):
 
 async def send_sys_message(app, channel_id: int, m_type: MessageType,
                            *args, **kwargs) -> int:
-    """Send a system message."""
+    """Send a system message.
+
+    The handler for a given message type MUST return an integer, that integer
+    being the message ID generated. This function takes care of feching the
+    message and dispatching the needed event.
+
+    Parameters
+    ----------
+    app
+        The app instance.
+    channel_id
+        The channel ID to send the system message to.
+    m_type
+        The system message's type.
+
+    Returns
+    -------
+    int
+        The message ID.
+    """
     try:
         handler = {
             MessageType.CHANNEL_PINNED_MESSAGE: _handle_pin_msg,

litecord/utils.py

@@ -39,6 +39,7 @@ async def async_map(function, iterable: Iterable) -> list:
 
 
 async def task_wrapper(name: str, coro):
+    """Wrap a given coroutine in a task."""
     try:
         await coro
     except asyncio.CancelledError:
@@ -136,7 +137,10 @@ def mmh3(inp_str: str, seed: int = 0):
 
 
 class LitecordJSONEncoder(JSONEncoder):
+    """Custom JSON encoder for Litecord."""
     def default(self, value: Any):
+        """By default, this will try to get the to_json attribute of a given
+        value being JSON encoded."""
         try:
             return value.to_json
         except AttributeError:
@@ -144,8 +148,7 @@ class LitecordJSONEncoder(JSONEncoder):
 
 
 async def pg_set_json(con):
-    """Set JSON and JSONB codecs for an
-    asyncpg connection."""
+    """Set JSON and JSONB codecs for an asyncpg connection."""
     await con.set_type_codec(
         'json',
         encoder=lambda v: json.dumps(v, cls=LitecordJSONEncoder),