New version of ApiOpenStudio Docker Dev

ApiOpenStudio Docker Dev has just been improved!

This is an update of the existing developer docker environment, geared towards developers of:

  • ApiOpenStudio Core
  • ApiOpenStudio Admin
  • Plugins
  • Connectors

It is not ready for full public release yet, because it implements the new ApiOpenStudio Docker Dev, which is still in pre-release.

However, it has some brilliant new features:

  • Uses Traefik as the reverse proxy (faster and better than Nginx reverse proxy).
  • All logging in a single place from Traefik and ApiOpenStudio (easier development processes).
  • A Makefile to simplify setup, initialisation and day to day usage (easier and faster setup and usage).
  • Example settings file, allowing you to get up and running in under 20 minutes (no need to worry about debugging your initial setup).
  • Persistent DB, stored in a docker volume for security, space-saving and portability.
  • Full HTTPS support and setup
  • Tested and working on all major OS’s:
    • Mac
    • Linux
    • Windows
  • The only requirements on the host OS:
    • Docker
    • Git
    • mkcert
  • Fully documented in the README, including FAQ for possible common issues and howto’s.

It will be a new major version release, will not be backwards compatible with the existing v1.x.x of apiopenstudio_docker_dev. Because of this, it will remain in the develop branch until apiopenstudio_admin_vue is ready for alpha release at the earliest.

You can checkout out code from the develop branch at https://gitlab.com/apiopenstudio/apiopenstudio_docker_dev/-/tree/develop, ensure that all repository checkouts are on the develop branch (for now).

This will enable a sneak preview on progress in the new Admin interface, that will include a drag n drop interface (for even faster and simpler development), and fully revised UX/UI.

ApiOpenStudio Production Docker video

We have released a new video on YouTube, demonstrating the speed and ease of deployment: spinning-up full running API and Admin installations on separate servers in under 30 minutes!

The ApiOpenStudio Production Docker video can be found at https://www.youtube.com/watch?v=iZ_Q81MhXUw.

This is a response to the discovery of an issue with the admin Docker image at the end of last week. The issue was resolved over the weekend and the docker images have been re-published.

ApiOpenStudio Admin production docker fixed

While preparing for yesterday’s seminar, it was found that the naala89/apiopenstudio_admin docker images were broken (the issue can be viewed here).

The good old fashioned “it works on my machine” came back to bite us, but the code has been updated and it is now loomed correctly into the GitLab pipelines. Many thanks to laughing_man77 for jumping on this so quickly!

Problem solved!

We’re pleased to announce that the issue has been resolved, and all tags have been re-uploaded to docker hub. So that was a really quick turn-around of 1 day (and we even had a chance to sleep on it before pushing it live).

Note: the only code that was affected was the https://gitlab.com/apiopenstudio/docker_images/apiopenstudio_admin_docker_prod repository. So if you have an existing checkout of the admin MVP code, this will not affect you.

Apart from the good news that full production images for ApiOpenStudio core and admin are fully working and tested, I also tested the released image on an existing server – and proved that the install time of admin is under 5 minutes!

Emerging new job titles in LCNC an indicator of profound change

What do we call the users of Low-Code or No-Code platforms that work for the IT dept?

Across the board, business and IT is seeing a transformation that is being driven by Low-Code and No-Code technologies. There is clear change in the job titles around development although the new terms have not yet stabilised.

Historically problems like this, are the indicator of genuine profound change!

Low-Code and No-Code is definitely not a replacement for developers or an IT department, and we will never be out of jobs (who will develop the Low-Code tools, who will design and build the surrounding complex architecture?). However, there are some great side-effects:

  • Business employees are learning the basics of
    • Development processes – the how-to and why they are beneficial, as Low/No-Code processes need to be constrained to stop ballooning and undocumented code.
    • Code and processes are more easily aligning with business processes.
    • Programmatic thought.

This is an exciting time, not only because we are seeing those at the forefront of the new technology explosion see their ideas come to fruition, but also because IT departments will see more understanding from business (as they learn more about IT as they onboard with the LCNC applications and processes), and this is driven by business and economics.

The language problem

Across many eras in IT there seems to be one thing that stands out about true – long lasting and permanent changes to the IT landscape break the language used in the current IT ecosystem. So any “language break-down” emerging is evidence of a strong of a probable trend being confirmed as profound change!

Some changes are fast and have “epoch moments”. Some are slow and creep up on us, leading us to say “this has always been known, but we never said it before in that way”.

Economists note that the one thing that stands out about genuine large scale change, is that the new activity will change what people do, how they are specialised and the job titles that describe it.

A good example of this is web-design. Initially we just had “web-designer”, before shattering that role into multiple disciplines, as we came to understand that improving the interfaces on a website was inherently complex, and architecture was only half the problem. We now have roles like (Sven Jenzer, 2023, Typical UX disciplines, [ONLINE] Available at: https://en.wikipedia.org/wiki/Parenthetical_referencing, Accessed 27 January 2023):

  • User Research
  • Information Architecture
  • Interaction Design
  • Usability Evaluation
  • Accessibility Evaluation
  • Visual Design

We need new job titles in development, so that Low-Code enabled developers can be classified in a way that reflects any extra value that they bring. While distinguishing them from citizen developers who, whilst they are not IT people, are doing ad hoc IT tasks to improve their role and also need a way of classifying their extra skills.

What is the solution?

Let’s look at the typical range of developer roles that we currently have:

  • IT Manager
  • Technical Architect
  • Technical Lead
  • Senior Developer
  • Developer
  • Junior Developer
  • Trainee Developer

We currently have no way of expressing the extra values that trained Low-Code developers and No-Code citizen developers bring. So, let’s break this down, essentially we are trying to express:

  • Experience level
  • Value to the business
  • Potential leadership role
  • Area of expertise

Using these taxonomies, we could potentially create new titles to express these, such as ApiOpenStudio Developer, Senior RapidAPI Developer, Junior Zapier Architect. However, this does beg the question, are users of Zapier or RapidAPI developers or architects? To be honest, not really, and they are definitely not architects. So this will not suffice.

I’m not posing a solution to this yet, and just want to open the floor to a problem that is likely to surface very soon, when those that have spent the time learning these skills want to express their new-found knowledge recognised by the business, or businesses want to find workers with the assets and skills that they want.

New Docker images for ApiOpenStudio!

The latest off the hot-press: We have finalised and published the docker creation pipelines. This now means that you will continually have the latest version of ApiOpenStudio Core and Admin images at your fingertips.

The images are very fast and easy to install and completely self-contained (server and all requirements, including Composer dependencies). That means that you do not need to install any infrastructure on a barebones server, except for Docker, config file and SSL certificates. See:

The scripts are configured to generate images from tags or any branches we want to release (this includes develop/master branches and any development branches that we deem necessary for testing).

Full documentation of the installation process is available on our wiki at:

The scripts are available at the GitLab docker repository (Core and Admin), and I’ve also taken a little time out to post a gist of one of the scripts that others may find useful (download a specified branch or tag from a GitLab or GitHub into a location) at https://gist.github.com/naala89.

ApiOpenStudio Introduction Seminar (Rescheduled)

Add to Calendar

ApiOpenStudio will be holding our first Live Event with our Developer/Founder, John

This event has been rescheduled to:

  • Friday 4th of November at at 2pm AEST.

The topics include: an introduction to the product and a run through of all the basics of the technology, and where you save time and money.

Followed by Q&A on the Product and the Business Concept.

No need to register as we will be live on YouTube. However if you would like to register, we are offering a free 90 minute support package to anyone who registers.

To register, fill in the contact us form on the home page.

Beta release is here!!!

We are excited to announce the official release of the Beta Version of ApiOpenStudio!

We have worked very hard on this release, and the release includes a lot of bug-fixes, enhancements and requested features. Here is a quick overview up the updates:

Beta release features

  • Updated the wiki:
  • Fixed issues in https://phpdoc.apiopenstudio.com.
  • Fixed caching:
    • Caching supports single and cluster cache servers, or on-server caching.
    • Added support for Redis.
    • Deprecated support for APC.
    • Implemented optional caching for individual processors.
  • Remote server and email outputs via plugins.
  • Created multiple plugins for remote output, see https://packagist.org/packages/apiopenstudio
  • Automated OpenApi document generation, supporting 1.x, 2.x, 3.x.
  • Ability to manage 3rd party packages and repositories within ApiOpenStudio.
  • Created two sample processor plugins
  • Implemented Fragments (process repeated calculations only once).
  • Added new processors to core:
    • JsonPath & XmlPath.
    • Cast.
    • Logic processors:
      • Do…While.
      • For…Each.
      • If…Then…Else.
      • Sequential.
    • Math.
    • Modules CRUD.
    • OpenApi CRUD.
  • Added more unit tests (all core functionality is now fully tested).
  • Added command line scripts (and API – so that it can be done using the GUI) to manage plugins.
  • Integrated the ApiOpenStudio CLI commands (install, update, module) with Composer.
  • Added a new page to the admin GUI to manage plugins.

We still have some known issues, and features that we want to build:

Tasks in the immediate roadmap

  • Begin the re-design and rebuild for the admin GUI with better UX and UI. This will include a drag ‘n’ drop interface to building resources.
  • Create a production docker for easier and secure installation.
  • Fix minor issues in the automated OpenAPI document generation
  • Add scanning for GET and POST variables to be automatically added to the OpenAPI documentation.
  • Mechanisms to upgrade or downgrade OpenAPI versions.
  • A mechanism to update OpenAPI if the domain is changed.
  • Add testing for PHP 8.1.
  • Add groups to permissions.

Free Support packages, for a limited time!

We now are seeing an exciting, 27% month on month growth in our install base of the Alpha version (ok “exciting” that’s a marketing term. We don’t normally use them, as that’s not us, but humour me – I think that’s what we are supposed to say).

We are particularly happy about this as we took the decision to launch ApiOpenStudio in an advanced form of Alpha. So we could get early feedback and build the best open source product possible. 

So to celebrate this, and the approaching Beta release, which we now estimate is now only 3 months away. 

We are offering a free 30 min support package to anyone who has downloaded the product. 

As this will support our early adopters, as well as give us the best possible feedback. The key to building the best product possible. 

This will be with our founder John (wherever possible), as we really want to get the maximum from this customer interaction.

So if you would like to take this up, please reach out to Matthew Greally on LinkedIn or the Contact Us form.

Major change to the ApiOpenStudio repository location

In order to implement pipelines and docker, with automated builds of docker images, the ApiOpenStudio projects have all been added to a new ApiOpenStudio group in GitLab.

This will enable GitLab pipelines to orchestrate pipelines across all of the projects as code is pushed and merged.

There was a dependency on this for upcoming tickets and tasks, so the tasks could not be delayed any longer. Because of this change, we have merged the develop branch to master branch, because this will update the wiki and phpdoc to reflect these changes.

However a new release tag for packagist has not been generated at this stage, becase we are only a few tasks away from beta release.

New changes available in the master branch:

  • GitLab CI pipelines now faster, (#118 – closed).
  • Wiki pages updated (#118 – closed & #115 – closed).
  • Fixed CI artefacts not being uploaded on failure (#117 – closed).
  • Logging now works on PHP8.0 as well as PHP7.4 (#111 – closed).
    • This involved deprecating Cascade, and creating a wrapper for the awesome Monolog package.
  • Implemented full JWT token authentication (#101 – closed).
  • Fix automated unit and functional tests (#110 – closed).
  • The entire project code has been updated to ensure all the latest PHPdoc and coding standards are passed.
  • Fixed Packagist for apiopenstudio_admin – sorry, this was my bad – it was a copy and paste error that went unnoticed.

Contributors and developers using the codebase

If you have a clone of the Gitlab repository, you will need to update your remote branch with the following command (assuming you have cloned with SSH):

git remote set-url origin git@gitlab.com:apiopenstudio/apiopenstudio.git

If you have a clone of the GitLab repository, you will need to update your remote branch with the following command (assuming you have cloned with SSH):

git remote set-url origin git@github.com:naala89/apiopenstudio.git

If you have forked the Gitlab repository, you can update the upstream URL:

git remote set-url upstream git@gitlab.com:apiopenstudio/apiopenstudio.git

The updated URLs

The new Group URL’s

The GitLab project URL’s

The GitHub mirror URL’s

Exciting upcoming features for the Beta release

  • Unit and Functional testing for PHP8.0 to ensure working across all contemporary PHP versions.
  • Composer 2.0 should be fine, but this should be tested before Beta release.
  • Swagger processor will be brought up to dat and fixed to allow importing and exporting of OpenApi documents.
  • Automated tagging and generation of an ApiOpenStudio Docker image

A Deep Dive into JWT Tokens in ApiOpenStudio Auth

Introduction

After a ton of work, over the next week or two we will be introducing JWT tokens for authentication in ApiOpenStudio.

This will:

  • Significantly increase the speed of resource requests.
  • Make individual transactions stateless.
  • Maintain the granular access rights to resources, based on the user’s access rights, the resource’s holding account & application, and of course the resource itself.
  • Ensure viability and ease for enterprise clients to use 3rd party authorisation services.

This article will take a look into the JWT specification, current practices and how it is used in ApiOpenStudio.

Rationale

While trying to optimise the authentication DB queries that are performed before the resource was processed, we came to the realisation that the queries were quite long and involved the joining of multiple tables and had several sub-queries so that the query could take into account our extensive range of access rights, like Administrators, Account managers, Applications Managers, Developers and Consumers. Although this would obviously be faster in a production environment, this was causing 1-2 seconds additional processing time to calls in development environments…

Because this had to be calculated every time a resource was called!

The solution was to introduce stateless authorisation, in the form of JWT tokens.

JWT is rapidly becoming the industry standard for authentication and JWT tokens have what are called Claims, which are individual name/value pairs within the body of the token and the token itself is encrypted and secure, which means that sensitive data can be securely included in the token and thus means that the user’s roles and permissions can be included as a claim and only need to be fetched once (during the JWT token GET call).

In addition, JWT tokens have a TTL, which means that we do not need to store a bearer token in the DB against each user (use that to fetch the user that each time a request is received) – because if the token is valid, then there is no need to fetch the user and check the bearer token TTL.

Here are some scenarios where JSON Web Tokens are useful:

Authorisation: This is the most common scenario for using JWT. Once the user is logged in, each subsequent request will include the JWT, allowing the user to access routes, services, and resources that are permitted with that token. Single Sign On is a feature that widely uses JWT nowadays, because of its small overhead and its ability to be easily used across different domains.

Information Exchange: JSON Web Tokens are a good way of securely transmitting information between parties. Because JWTs can be signed—for example, using public/private key pairs—you can be sure the senders are who they say they are. Additionally, as the signature is calculated using the header and the payload, you can also verify that the content hasn’t been tampered with.

auth0.com, JSON Web Token Introduction – jwt.io, viewed 1 September 2021, https://jwt.io/introduction.

How does JWT work?

Pronunciation

Before we start, for a bit of fun I’d like to set the record straight. I’m not one of those boring “absolutists” who insist that GIF should be pronounced “JIF”, even though the original creator of the tech obviously could not spell when he declared that was the pronunciation. But many people are driving me bananas, by mispronouncing JWT: just pronounce it as “JOT”, it’s also easier to say than the most common variant “Jay-Dubbyah-Tee” (this is not a relative to a former US president).

The suggested pronunciation of JWT is the same as the English word “jot”.

Jones M, Microsoft, Bradley J, Ping Identity, Sakimura N, NRI 2020, JSON Web Token (JWT), Internet Engineering Task Force (IETF),viewed 1 September 2021, https://datatracker.ietf.org/doc/html/rfc7519#section-1.

JWT overview

Now for the technical stuff…

JWTs represent a set of claims as a JSON object that is encoded in a
JWS and/or JWE structure. This JSON object is the JWT Claims Set.
As per Section 4 of RFC 7159 [RFC7159], the JSON object consists of
zero or more name/value pairs (or members), where the names are
strings and the values are arbitrary JSON values. These members are
the claims represented by the JWT. This JSON object MAY contain
whitespace and/or line breaks before or after any JSON values or
structural characters, in accordance with Section 2 of RFC 7159
[RFC7159].


The member names within the JWT Claims Set are referred to as Claim
Names. The corresponding values are referred to as Claim Values.
The contents of the JOSE Header describe the cryptographic operations
applied to the JWT Claims Set. If the JOSE Header is for a JWS, the
JWT is represented as a JWS and the claims are digitally signed or
MACed, with the JWT Claims Set being the JWS Payload. If the JOSE
Header is for a JWE, the JWT is represented as a JWE and the claims
are encrypted, with the JWT Claims Set being the plaintext encrypted
by the JWE. A JWT may be enclosed in another JWE or JWS structure to
create a Nested JWT, enabling nested signing and encryption to be
performed.


A JWT is represented as a sequence of URL-safe parts separated by
period (‘.’) characters. Each part contains a base64url-encoded
value. The number of parts in the JWT is dependent upon the
representation of the resulting JWS using the JWS Compact
Serialization or JWE using the JWE Compact Serialization.

Jones M, Microsoft, Bradley J, Ping Identity, Sakimura N, NRI 2020, JSON Web Token (JWT), Internet Engineering Task Force (IETF),viewed 1 September 2021, https://datatracker.ietf.org/doc/html/rfc7519#section-3

What does this mean?

A token is comprised of 3 parts, the header, payload and signature.

The header of the token defines the cryptography applied to the payload, and thepayload is a JSON structure, that is encoded in JWS (Base64 encoded) or JWE (encrypted).

JWS is less secure since it is only a Base64 encoded JSON, and this means it is not suitable in our case. Because the token will carry authentication data for access to the resource and Base64 encoding is different to encryption – it is relatively trivial to decode.

JWE is more suitable to our case, because the JSON payload is encrypted (as defined in the header), and the JWT token keys are stored securely on the on the ApiOpenStudio server, so only the API server can decrypt the body.

There are multiple encryption standards available, and in our case we are using the fantastic lcobucci-jwt library, which is sponsored by one of the leading Authorisation services: Auth0. This provides support for many, many symmetric and asymmetric algorithms.

We have not implemented symmetric algorithms (these are less secure and are the same key for encryption and decryption), so with asymmetric algorithms, the public key can be used by any clients, and the public/private keys are stored in a secure location on the ApiOpenStudio server.

JWT token structure

In its compact form, JSON Web Tokens consist of three parts separated by dots (.), which are:

  • Header
  • Payload
  • Signature

Therefore, a JWT typically looks like the following.

xxxxx.yyyyy.zzzzz

Header

The header typically consists of two parts: the type of the token, which is JWT, and the signing algorithm being used, such as HMAC SHA256 or RSA.

For example:

{
  "alg": "RSA256",
  "typ": "JWT"
}

Then, this JSON is Base64Url encoded to form the first part of the JWT.

Payload

The second part of the token is the payload, which contains the claims. There are three types of claims: registeredpublic, and private claims.

  • Registered claims: These are a set of predefined claims which are not mandatory but recommended, to provide a set of useful, interoperable claims. Some of them are: iss (issuer), exp (expiration time), sub (subject), aud (audience). Notice that the claim names are only three characters long as JWT is meant to be compact.
  • Public claims: These can be defined at will by those using JWTs. But to avoid collisions they should be defined in the IANA JSON Web Token Registry or be defined as a URI that contains a collision resistant namespace.
  • Private claims: These are the custom claims created to share information between parties that agree on using them and are neither registered or public claims.

An example payload could be:

{
  "iss": "my.apiopenstudio.com",
  "sub": "1234567890",
  "name": "John Dory",
  "admin": true
}

The payload is then Base64Url encoded to form the second part of the JSON Web Token.

Signature

To create the signature part you have to take the encoded header, the encoded payload, a secret, the algorithm specified in the header, and sign that.

The signature is used to verify the message wasn’t changed along the way, and in the case of tokens signed with a private key, it can also verify that the sender of the JWT is who it says it is.

Using JWT tokens

The finalxxxxx.yyyyy.zzzzz token is sent in the request as a bearer token in the request header, e.g.:

Authorization: Bearer <token>

When the request is received by the API server, it will first confirm that the token is valid, checking it can be decrypted, the issuer, expiry date and mandatory claims.

If this all passes, the processing can continue, otherwise a 401 error response is sent and the client will need to generate a new token, using the provided core token request (auth/token) and resend the request with the token that was received.

From a processes POV, this is exactly the same as before. However, as mentioned, we are using asymmetric encoding, so we can include data in the token payload that means that ApiOpenStudio does not need too fetch user data in order to validate the user’s permissions against the resource. That only needs to be fetched once – when the token is generated.

Minor issue (caveat)

The original authorisation tokens were stateful, i.e. the token was stored against the user, along with the TTL for the token. This meant that if a user was banned, deleted or made inactive, they would instantly not be able to make any further API requests, due to either the users not being present/active anymore, or the token was no longer valid.

JWT tokens are stateless. This means that if a user has a valid token, they can still use it until it expires regardless of whether they are made inactive or deleted (they would only be prevented from fetching a fresh token).

This can be mitigated by setting a global jwt_life of less than 1 hour (the default), however this needs to be balanced against the increase in requests, since a lower token TTL will lead to more requests due to tokens frequently passing their expiry date more frequently.

Another mitigation for extreme cases can be to add the IP address of the client to the blacklist – this will prevent all future calls from that location, immediately.

3rd party authorisation integration

Because the token requires certain client data to be present, the user details and roles will need to be accessible by the authorisation provider, so that they can generate a valid body. Thankfully, most reputable providers allow you to upload these details to your account with them, and also provide ways to ensure that these details are always current and up to date.

You will need to ensure that the following mandatory claims:

  • iss – JWT issuer (your auth provider)
  • aud – permitted for (your api)
  • iat – JWT issued time
  • exp – JWT expiry time

The following ctsom claims are also included

  • uid – user ID
  • roles – complete list of roles and accounts/applications that the user is associated with

The roles object

This is in the JSON object format of:

roles: [
    {
        "role_name": <role_machine_name>,
        "accid": <account_id>,
        "appid": <application_id>
    }
]

For example:

roles: [
    {
        "role_name": "administrator",
        "accid": null,
        "appid": null
    },
    {
        "role_name": "consumer",
        "accid": 34,
        "appid": 5
    },
    {
        "role_name": "developer",
        "accid": 34,
        "appid": 5
    },
    ....
]

Note that:

  • “administrator” does not require accid or appid
  • “account_manager” does not require appid

Summary

We’re really excited to be implementing this technology, seeing the decrease in resource processing time, increasing the security of the API, and making it even easier for enterprise scale users to implement ApiOpenStudio on a large scale.

I’ll be honest with you too, it was really good fun to implement, and we totally got a total nerd-on, doing the research and coding for this!

WP Twitter Auto Publish Powered By : XYZScripts.com