The Community Solid Server: Supporting Research & Development in an Evolving Ecosystem

Introduction

TODO: the socio-economic problem Solid is solving (see my blog posts for inspiration)

Solid [1] is many things. I think that conventions are not enough; technical specifications to achieve the aforementioned goal? At the core it is a set of API conventions to facilitate built upon the Linked Data Platform [2] (LDP) specification. Before describing what it adds, explain the needs of why this is added It adds, among other things, components such as identity, authentication and authorization. But that is just the server component.

Besides the servers, there are also the Solid applications, which are client applications that effectively use Solid servers as a backend through the Web. The idea there is that multiple applications can independently make use of the same server, while still interacting with the same data. Yes, make the point a bit more explicitly: the apps conform to the spec, the server conforms to the same spec, so any app works with any server. The browser/Apache comparison might be useful; here or already earlier. This way a person only needs to store their personal data in a single place which can then be reused by all applications that have need of this information.

Solid is deeply a community effort: all layers influence each other. Which layers? The server API and specification naturally influence how the applications have to be developed. On the other hand, the actual experiences of application developers and how they experience interacting with the server is vital in ensuring a healthy development of the environment.

Due to the open-source nature of Solid development, both in available tooling and discussions being had, everyone can cooperate to improve what is available, which again emphasizes the community aspect of Solid.

The Community Solid Server was built from the ground up to support this community aspect. Starting from the design phase, the idea was always that it could be used by a multitude of users: ranging from people who want try out this Solid thing, to application developers that want to test out their new app against a certain setup, to backend developers that want to set up Solid servers with specific requirements.

Requirements

See comunica paper for inspiration: https://comunica.github.io/Article-ISWC2018-Resource/

As mentioned in Section 1 the goal of the Community Solid Server is to provide a wide spectrum of possibilities for everyone that wants to get involved with Solid, from starting hobbyists to Solid-focused research labs, all this while still remaining maintainable for future work.

There are three different axes we wanted to cover during the design of the server, which we will discuss below.

I see 4 🙃

I wonder if we should have a stakeholders discussion. What do researchers and developers aim to achieve?

Perhaps spec writers are stakeholders too.

Evolving Solid specification

The Solid specification is still a draft specification, meaning it can and will change in the future. A consequence of this is that any Solid server implementation that wants to stay relevant has to be flexible enough, so it can be updated once spec changes occur. Likewise, a modular server can be used to inform the specification: if there is any doubt, a new module can quickly be added to assess the impact of a change. That then ties into research, where we can look at designing completely different APIs and algorithms, and see how it behaves

Research

Solid is still growing as an ecosystem, this includes the research that investigates all the possibilities of using Solid as a core part of a system. This sometimes requires quickly setting up thousands of Solid servers for automation and having flexibility in their configuration such as what storage method is used, how authorization works, or how user accounts are managed, for example. A specific class of research (and perhaps this also ties into the next point) could be intermediaries, i.e., not just the clients and servers we’re talking about today

Server Customization

No Solid server is ever going to fully support everything a user wants: there are infinite possible combinations of features that could be required. For this reason we wanted to ensure that the implementation allows for easy extension, so that if, for example, a user wants the server to use their own custom backend, they do not have to rewrite a significant part of the server logic.

Application Development

There are many things that have to be checked and kept in mind when developing a new Solid client application. As mentioned before, it is vital that the application correctly makes use of the Solid API, but besides that there is also a need for test users, testing different authorization situations, managing different error responses, etc. This requires testing against an actual server that provides a quick setup and teardown, and can easily emulate different situations.

Existing implementations

Think this makes more sense here instead of in related work?

I don’t want to talk too badly about other tools :D

use more complex request (PATCH) to explain why CSS handles this better? - NSS would not be able to add ACP instead of ACL?

LDP is just an API, could also have SPARQL endpoint in addition?

Some existing LDP servers include Apache Marmotta, Virtuoso, and Trellis.

There are also already several existing Solid servers, such as Node Solid Server, Enterprise Solid Server, TrinPod, Reactive-Solid, and several others.

While they each have their own advantages and disadvantages, none of these fulfils the requirements that we set up before though. Specifically the customization and flexibility requirements that we have, are never fulfilled.

Related Work

Many more references?

Linked Data Platform

Linked Data Platform [2] (LDP) is a set of rules to provide a RESTful API to read/write Linked Data resources on the Web. It introduces several concepts such as using URIs to identify resources and using RDF standards to provide information and data.

Solid

The Solid [1] project is a combination of specifications with a focus on data ownership and data interoperability. At its core is the Solid Protocol specification which determines how Solid servers should behave. It is built upon the LDP specification and has a stricter and more well-defined expectation for each of the CRUD operations.

Components.js

Components.js [3] is a Dependency Injection framework built specifically for TypeScript projects. It is non-invasive in that all of its configurations happens outside the source code in external configuration files, which describe how classes are linked to each other and which parameters they require. These descriptions are done in RDF, which means that those configuration files are valid RDF serializations. Usually JSON-LD is used.

OIDC

To handle authentication, the server implements the Solid-OIDC specification, which builds upon the OAuth 2.0 [4] and OpenID Connect Core 1.0 standards. These are two widely adopted standards, and the above specification extends them to solve problems specific for the Solid ecosystem.

Architecture

Section 2 explained that there is a strong need for flexibility in the server. We want it to be possible for anyone to set up the server tailored specifically to their needs. We enabled this by making the codebase completely modular: classes are completely independent of other classes’ concrete implementations, and only define which interfaces they expect. This allows anyone to easily swap out any implementation without impacting other components, as long as they follow those same interfaces.

Components

So what I’m missing a bit below are the simple truths: you start the server with a config file, that is JSON-LD. These configuration files can be mixed and match to start slightly different servers: great for R & D. To support that, we did… It’s all in there, but in different places, and it’s hard to get it out unless you exactly know what you’re looking for—and only we as authors do. Note that this can perhaps be mitigated easily when moving Section 5 before this section, as I explain there.

To combine the classes and set up their dependencies, we make use of a Dependency Injection (DI) framework, which injects actual implementations into the classes. This way the actual class chains are configured externally to our implementations, making these much easier to support changes, as required for research and development. We use the Components.js [3] DI framework, which uses declarative RDF configuration files to link everything together, as detailed in Section 5.

I think every paragraph here should go to a point. What does it mean concretely for developers and/or researchers?

Due to the nature of the DI-based architecture, every component is mostly unaware of own role in the grand scheme of things. For example, there is a component that converts authentication headers into a usable identifier, not knowing which component will make use of it. There is a component that converts thrown errors to serializable output, and another simply takes incoming RDF serializations and converts them to quad objects.

The narrative structure of this paragraph doesn’t make sense to me; I don’t get the point/conclusion I’m supposed to draw. We don’t bring home the point whether the unawareness is desired or undesired, and the examples show that they exist, but do not list any consequences. Can we update the explanations and examples to make sure that we’re going to a point?

The combination of this architecture, combined with the DI framework, brings many advantages for both developers and users. I believe you, but you gotta go concrete 🙂 Which advantages? How so?

Since the Solid specification are still evolving, it is important that the impact of changes impacts a minimal number of components. Note that the previous sentence is a non-sequitur. The reasoning is (I think): the spec is still evolving, we want to reduce the impact of changes, we do this by organizing the architecture such that a small numbers of components need changes. (Do we have examples of this?) On the one hand, this makes it much easier for developers to keep the server in line with the specification. On the other hand, this also reduces the chances of breaking any extensions that might have been created for the server. But how do we achieve this?

Components.js does not require the components it links together to be in the same repository. This means anyone can easily extend the features of the server by writing their own modules. This helps researchers with the comparison of multiple ideas; only a configuration change is required to set up different versions of the server. This is highly useful when doing research to evaluate different aspects, or even as part of discussions on the Solid specification to immediately see certain suggestions in practice.

While it is possible to extend the server with custom components, the reverse is also true: individual components of the server can be reused. All the components are exposed through the project, so it is possible to include only a few of them specifically if preferred. The configurations provided with the server can also be reused, allowing the reuse of partial blocks without having to reconfigure them completely.

High-level Architectural View

Should cover some parts of how a Solid server works in the related work as we don’t want to cover all of it here?

Yes, absolutely! That can be half of relwork actually.

Should also use references instead of just links.

The goal of the server is to accept incoming HTTP requests and send a correct response based on what is defined in the Solid specification. The specification consists of several core parts. The community server supports these by having different components for each of them. These can then be consecutively applied to the incoming request to reach the final result, with each of them potentially stopping the request in case of an invalid request. Below we cover some of these major core components.

Authentication

Authentication in Solid is part of the Solid-OIDC specification. The authentication block of the server will parse the relevant headers of an incoming request to identify the agent calling the server. It will output the correct identifier to be used by other components.

Authorization

For authorization, the Web Access Control specification is used. There are multiple components at work here as handling this requires several steps. The sever has to determine which permissions are required based on the kind of request. It also has to determine which permissions are allowed on the target resource. A request is only valid if required permissions are allowed.

Solid Protocol

The main Solid specification is the Solid Protocol. It is based on the Linked Data Platform [2] specification and determines how different possible HTTP methods should be interpreted by the server. There are many relevant components here as there are many possibilities here, but the end goal is always to perform the necessary data action and return the result.

Reductive Request Processing

As we mentioned before, all the components in the server solve a specific problem and are mostly unaware of the grander scheme of what is going on. Requests to the server are solved by letting each component handle a small part of the problem, so the next component can continue and do the same, until it has been completely solved.

TODO: PRETTY PICTURE
- HTTP Request -- Request Parser --> Operation
- HTTP Request -- Credentials Extractor --> Credentials
- Operation -- Modes Extractor --> AccessModes
- Credentials, Operation -- Permission Reader --> Permissions
- Credentials, Operation, Permissions -- Authorizer --> Reject/Accept request
- Operation -- Operation Handler --> Response Description
- Error -- Error Handler --> Response Description
- HTTP Response, Response Description -- Response Writer --> Write HTTP Response

Listing 1 shows a simplified overview of how a request gets resolved. It starts as an HTTP request and ends as the output is written as an HTTP response. The steps are as follows:

1. The request is parsed into an easy-to-use Operation object containing all the parsed essentials of the request.
2. We extract the credentials from the request. Generally this will either be blank or a signed version of the WebID identifying who is doing the request.
3. From the Operation we extract which CRUD permissions are required to resolve the request.
4. The permission reader determines what the request owner is allowed to do on the target resource.
5. The authorizer determines if the request can proceed based on the output from step 3 and 4.
6. The Operation handler resolves the Operation and generates a ResponseDescription, containing everything needed to write a valid response.
7. In case any of the previous steps failed, a ResponseDescription will be generated based on the error thrown.
8. The ResponseDescription is used to write a response.

Data storage

Solid does not specify how data should be stored, only how it should be returned. Internally we have done the same by abstracting data access with a Resource Store. This allows us to have different stores for different storage methods, such as in-memory, file-based or with a SPARQL endpoint. This interface has functions corresponding to all the CRUD requirements. In the previous subsection, the Operation handler of step 6 calls the corresponding function based on the HTTP method.

TODO: PRETTY PICTURE
Operation Handler
-> Locking Store
-> Patching Store
-> Converting Store
-> Data Accessor Store

Perhaps some of the new documentation is useful here?

In practice, the Resource Store is actually several store implementations all chained together, as can be seen in Listing 2. Each store again handles a specific a part of the complete behaviour that is expected from the store:

• The Locking store prevents multiple operations from writing to a resource at the same time.
• The Patching store handles `PATCH` requests.
• The Converting store converts representations to support content negotiation.
• The Data Accessor store supports LDP behaviour by calling a Data Accessor class, which is a simple interface to support a specific storage method. E.g., file based, memory based, etc.

Example Request Runthrough: PATCH

One specific example of how the server makes use of smaller independent tools to solver a bigger problem can be seen in how it handles PATCH requests. Currently, Solid servers generally only accept PATCH requests that contain a SPARQL UPDATE body to edit RDF resources. Add N3 Patch, link to spec

One solution for this would be to support these queries in the same component that also handles storing the data, seeing as we have direct access to the data there. The disadvantage is that such an implementation would be required for every different storage method used, thus also providing extra work for developers that want to support a new storage method on the server. Bring home the impact on researchers and developers

The CSS handles this differently. As can be seen in Listing 2, it has a component specifically for handling PATCH requests. This store first checks of the next store supports a PATCH by itself, as could for example be the case if the storage method is a SPARQL endpoint. But if this is not the case, this store provides a fallback method:

1. Create a new internal request to acquire the data, using content negotiation to request triple data.
2. Execute the SPARQL UPDATE query on the received triple data in memory.
3. Create a another internal request to write the resulting data to the resource, again using content-negotiation to convert back to the original media type.

The converting component in this story is unaware that a PATCH request is using it, it simply knows that it first gets a request to convert an RDF serialization to triples, and then another request to convert triples into a serialization again. Likewise, the backend storage only knows that it first has to return a data stream, and afterwards has to overwrite the resource data. On the other hand, the patching component does not know what the original data serialization was, or how it was stored; it can only apply a SPARQL UPDATE query on a set of triples.

conclude

Configuration

Suggestion: move this before architecture. Then we can talk about this on a high level: why are different configs so interesting? What can we do with them? Then the architecture section can explain how the configs actually work. I.e., use the configs as a starting point, as a necessity, and then the things the architecture needs to do to support this.

As mentioned in Section 3, the server makes use of the Dependency Injection framework Components.js [3] This allows us to combine all the independent components discussed in Section 4.

It is a framework with much flexibility and options, but it does have a steep startup curve before fully understanding how configuration works. To this end we took several steps to make configuration as easy as possible for new users.

Default configurations

Due to the external nature of the configurations, Components.js allows us to provide many different versions of the server to the users.

The server comes bundled with several default configurations that can be used out of the box. These include different backends, such as file or memory based, and examples on how to configure more complex features.

These default configurations cover the main use cases of many users, and those who do want to configure a different experience can often achieve this by only making minor adjustments.

Feature options

Since RDF is used, it is possible to split the configuration up over multiple files and then import them into a single file. We strongly make use of this import behaviour to hide most of the configuration complexity from new users. Specifically, we made it so a user can choose specific features based on the files being imported. For example, the only difference between a configuration to set up a server with a memory backend compared to one with a file backend is that the first one imports /storage/backend/memory.json and the second one imports /storage/backend/file.json. This is just one of the imports that users can modify to change behaviour. To help with the choices there, there is documentation explaining all the available options.

Configuring extensions

When someone wants to develop a new component for the server, adding Components.js configuration to link it correctly will be a requirement. Components.js also requires configuration description files for every class, describing the class parameters, but the server is set up to generate those automatically using Components-Generator.js, saving developers much work. Besides that, the many configurations included with the server should help as examples of how to link components.

Walk us through some extensions; walk us through some recipes and briefly what they do.

Usage & Impact

Repository

Update numbers when finished

At the time of writing, the repository has 288 stars and 72 forks. The corresponding gitter chatroom has 32 people.

Maintenance

During the development of the server we have always focused on making sure the code bases remained of high quality. One way we did this is by requiring the unit tests to always have 100% code coverage on all code in the project. While this is not immediately an indication of everything working as intended, it does make sure that a developer checks that new classes output data as expected.

Another requirement is that all new code needs to be added through pull requests, which always require at least one code review before being able to be merged. This means that all code is seen by multiple people before being added.

Besides the unit tests, we also have extensive integration tests covering the larger parts of the server. These, for example, test all the entire Identity Provider procedures and all the supported LDP actions.

Projects

Some projects that other people have already made with the server:

• Reading calendar data using a Solid server: https://github.com/KNowledgeOnWebScale/solid-calendar-store/
• Operating Philips Hue lamps through a Solid server: https://github.com/RubenVerborgh/solid-hue/
• Data-Kitchen, a desktop app combining local files and Solid pods: https://github.com/solid/data-kitchen/
• Solid-Redis, a component for the server to use Redis as data storage: https://github.com/comake/solid-redis

use.id is an example of a commercial product that was built using the server.

Make sure this URL (and many others) also appear in the printed version.

And all apps that work on the server? Do you mean Solid client apps? Will have to investigate which ones are up to date enough to work. Media Kraken and Mashlib are at least 2.

Do we explicitly have to say which projects are related to our lab?

Future Work & Conclusions

We have achieved our initial goal of creation a server that has uses for a wide target audience, as can be seen by the interest of both hobby and expert users.

In the future we want to keep making sure the server adheres to the Solid specification, but also keep extending its functionality so it becomes both more user-friendly and has a wide array of different features it supports.

One of the big challenges for the server is versioning. We follow the semantic versioning guidelines to make sure we do not break any server installation. For each new release, notes are added indicating both the new features and how configurations of older versions can be upgraded to the newer version. Since all classes and configurations are part of the public API of the package, most changes immediately require a major version increase, which causes a faster upgrade cycle.

Due to the modular nature of the components, it is not required for all new features to be added to the main repository. This makes it possible for companies to develop the features that they require and then link them to the main server, thereby becoming the maintainers of that specific feature, which could be part of their business model.