API Insights
API Insights is a tool to enable organizations to manage versioned API specifications (Swagger 2.0/OpenAPI Spec 3.x) for services. It also does static analysis of API spec files for compliance against REST API best practices guidelines, document completeness, inclusive language check and runtime API drift from documented spec. To help API consumers and developers, API Insights service also supports generating an API changelog including identification of backward compatibility breaking changes between 2 versions of API spec files.
API Specifications Challenges
- As the number of services increases, no common place for storing versioned API specs.
- Inconsistency in API specifications across teams. Makes it difficult for API consumers that integrate across multiple APIs.
- API changes across versions could result in breaking backward compatibility.
- Lack of consistent documentation of API changes across multiple releases.
Solution
API.Insights-Demo.Video.mp4
- API Insights service enables storing of multiple versions of released (& release candidate) API specifications.
- Validate & Score API Spec against guidelines:
- API Insights REST guidelines
- Guidelines are tested using API guidelines linter
- API Document completeness
- Inclusive Language Ruleset
- API Drift analyzer (Integrate with APIClarity to identify Zombie and Shadow APIs)
- API Security analyzer (Future: Integrate with Panoptica to enable API Security analyzer)
- API Insights REST guidelines
- API spec diff across multiple versions/revisions
- Identify and alert on backward compatibility breaking changes.
- API Insights CLI to enable running spec analyzer as a part of API spec CI/CD or local commit pipeline.
User Flow and Architecture
User Flow
- Developer or Tech Lead can upload the API Specification and subsequent revisions:
- Commit new version/revision of spec in GitHub repository.
- CI/CD pipeline with specs analysis against guidelines & generate report/score.
- On GitHub release tag, new version/revision of spec will be uploaded to the API Insights service by CI/CD task.
- Multiple API specs across products/services can be managed in the API Insights service.
- On new spec upload, preconfigured analyzers will run on spec in background.
- User can go API Insights UI to view:
- Analyzer score and issue listing with trends across releases.
- Detailed report with severity, line number and remediation recommendations.
- Users will be able to see a summary of all API changes (New, Modified, Removed & Breaking) and will be able to see the detailed spec diff by clicking on each changed item.
- Integration with APIClarity & Panoptica will allow:
- Security and Compliance users to get reports on Zombie & Shadow APIs
- Reconstructed OAPI for missing specs
- Security Analysis of API
Related Projects and resources
- API Insights VSCode Extension
- API Insights OpenAPI Ruleset
- Inclusive Language Ruleset
- API Insights site
- API Insights Doc
Getting Started
This repo contains a Helm based deployer that can be deployed in a local Kubernetes cluster setup using like Rancher Desktop, minikube etc. The detailed instructions are found here.
Development setup
Build and start UI & backend services using Docker Compose
docker-compose up
Once Docker Compose is up, UI and be access at http://localhost:8080
- To run the API service natively outside docker, refer to api/README.md
- To run the UI natively outside docker, refer to ui/README.md
Note: Docker-compose requires installing Rancher Desktop or licensed 'Docker Desktop'.
Contribution
We welcome contributions, please find details in CONTRIBUTING.md
backend: data compression feature.
Utilizes GORM's hooks (https://gorm.io/docs/hooks.html) to compress (before saving) & decompress (after querying) large-sized data (size threshold determined by config
start-data-compression-at-bytes
).start-data-compression-at-bytes
: Value in bytes at when to compress blob; default=3145728,0=compress all,-1=no compressionUsed for Spec.Doc, SpecAnalysis.SpecAnalysisResult.RawResult, SpecDiff.SpecDiffResult.RawResult.
change mysql to 5.7.40 to avoid sort buffer oom
This is to avoid sort buffer oom in mysql 8.x. Please note, If you meet some issue to docker compose up this version of mysql. It probably because the old data of mysql 8.x. You need to remove them by: docker compose down --volumes
Show the weight of each analyzer and its score
In the current report page, we can only see a list of analyzers used on a spec. However, it is not clear how much weight was applied and how the score was calculated for each analyzer.
To address this issue, the UI will need some adjustments to display these information.
Nginx body size
increase nginix client_max_body_size so that larger specs can be uploaded successfully. Fixes: https://github.com/cisco-developer/api-insights/issues/50
backend: remove tmp diff files
Remove diff files that are temporarily created during diff execution.
These files were originally retained for testing purpose, but can now be cleaned up.
Minor README enhancements
Fixed some typos, and made some enhancements for clarity & consistency.
For example, use consistent form of
CI/CD
, use accurate branding terms likeOpenAPI
instead ofOpen API
,GitHub
instead ofgithub
,Docker Compose
instead ofdocker-compose
, etc.NGINX issue while deploying in Openshift
After deploying helm chart in Openshift platform , facing below error. Please help on resolving below error.
sed: can't create temp file '/etc/nginx/nginx.confXXXXXX': Permission denied 2022/12/12 07:00:17 [emerg] 8#8: mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied) nginx: [emerg] mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied)
For this issue, Stackoverflow suggested below mentioned solution to rebuild image. I have to build and package UI application. Please let me know UI setup details.
https://github.com/nginxinc/docker-nginx/issues/417
Update CLI
--fail-on-error-rule
(defaults to exit code 0 w/o the flag)Diff & breaking change reference object preview
Scenario: When showing diff between 2 versions of the specs doc or highlighting backward compatibility breaking changes, sometimes changes are not visible in the API Path request or response object but changes are in reference objects. Reference objects are typically defined under components (OAS3) or definitions (swagger2). For enabling users to do a more meaningful analysis of spec diff, API Insights UI needs to support allowing showing diff of referred components in the context of API Path.
Proposed Solution:
UX should be improved to allow users to see the diff between a referenced object in the API Path context. For example in the sample case "#/definition/ListResponse" can be expanded to see the diff between ListResponse objects in 2 versions.
New API should be exposed by the API Insights service that will return reference object GET /v1/apiregistry/services/{id}/specs/{specID}/references
UI while showing diff should detect "$ref": "#/components/" in the diff JSON/YAML & enable view reference functionality as per design in step 1.
Large API spec file upload failing.
Issue: When uploading a large spec file for a service, upload failed with an error request entity too large. See below snapshot
Expected Behavior: Upload spec should work and spec should get analyzed.