Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 18 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ The Rhesis Java SDK empowers developers to programmatically access curated test
- [Generating Custom Test Sets](#generating-custom-test-sets-%EF%B8%8F)
- [Examples](#examples-)
- [Test Execution](#test-execution)
- [Analytics & Stats](#analytics--stats)
- [Insights](#insights)
- [About Rhesis AI](#-about-rhesis-ai)
- [Community](#-community-)
- [Hugging Face](#-hugging-face)
Expand All @@ -65,7 +65,7 @@ The Rhesis Java SDK provides programmatic access to the Rhesis testing platform:
- **Access Test Sets**: Browse and load curated test sets across multiple domains and use cases
- **Generate Test Scenarios**: Create custom test sets from prompts, requirements, or domain knowledge natively using local LLM models
- **Execute Test Runs**: Trigger test set runs against your endpoints in parallel or sequential mode
- **Analytics & Stats**: Typed stats for test runs and test results — pass rates by metric, requirement, category, topic, and timeline trends
- **Insights**: Aggregation queries for test runs and results — pass rates by requirement, category, topic, with date range filtering
- **Manage Metrics & Tests**: Add/remove metrics on test sets, associate/disassociate tests, rescore previous runs
- **Seamless Integration**: Integrate testing into your Java CI/CD pipeline and development workflow
- **Comprehensive Coverage**: Scale your testing from dozens to thousands of scenarios
Expand Down Expand Up @@ -226,8 +226,8 @@ Your API key will be in the format `rh-XXXXXXXXXXXXXXXXXXXX`. Keep this key secu
```java
import ai.rhesis.sdk.RhesisClient;
import ai.rhesis.sdk.entities.TestSet;
import ai.rhesis.sdk.synthesizers.Synthesizer;
import ai.rhesis.sdk.synthesizers.GenerationConfig;
import ai.rhesis.sdk.synthesizers.Synthesizer;
import java.util.List;

public class Main {
Expand All @@ -238,19 +238,27 @@ public class Main {
RhesisClient client = RhesisClient.builder()
.apiKey("rh-your-api-key") // Get from app.rhesis.ai settings
.build();
RhesisClient.setDefault(client);

// Browse available test sets
List<TestSet> testSets = client.testSets().list();
for (TestSet testSet : testSets) {
System.out.println(testSet.getName());
System.out.println(testSet.name());
}

// Generate custom test scenarios
Synthesizer synthesizer = new Synthesizer("Generate tests for a medical chatbot that must never provide diagnosis");
// Generate custom test scenarios with a name
GenerationConfig config = GenerationConfig.builder()
.generationPrompt("Generate tests for a medical chatbot that must never provide diagnosis")
.testSetName("Medical Chatbot Safety Tests")
.requirements(List.of("Refuses diagnosis", "Recommends professional consultation"))
.build();

TestSet generatedTestSet = synthesizer.generate(10);
TestSet generatedTestSet = new Synthesizer(config).generate(10);
System.out.println("Generated Tests:");
generatedTestSet.tests().forEach(test -> System.out.println(test.prompt()));

// Push to the platform
client.testSets().create(generatedTestSet);
}
}
```
Expand Down Expand Up @@ -282,9 +290,9 @@ Looking for more detailed examples? Check out the full [Examples README](src/tes
**Test Set Management**
- [Test Set Metrics](src/test/java/ai/rhesis/sdk/examples/TestSetMetricsExample.java) — List, add, and remove metrics; associate and disassociate tests

**Analytics & Stats**
- [Test Run Stats](src/test/java/ai/rhesis/sdk/examples/TestRunStatsExample.java) — Overall summary, status distribution, most-run test sets, timeline, filtered queries
- [Test Result Stats](src/test/java/ai/rhesis/sdk/examples/TestResultStatsExample.java) — Pass rates by metric, requirement, category, and topic; per-run summaries and timeline trends
**Insights**
- [Test Run Insights](src/test/java/ai/rhesis/sdk/examples/TestRunStatsExample.java) — Run counts by status, date range filtering with months and start/end dates
- [Test Result Insights](src/test/java/ai/rhesis/sdk/examples/TestResultStatsExample.java) — Pass rates by requirement, category, and topic; date range queries and failed ID retrieval

You can run any example from the command line using Maven. Make sure your `RHESIS_API_KEY` is set in your environment (it will be automatically picked up from a `.env` file at the root of the project if one exists):

Expand Down
7 changes: 7 additions & 0 deletions src/main/java/ai/rhesis/sdk/clients/TestSetClient.java
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,13 @@ public List<ai.rhesis.sdk.entities.Test> getTests(String id, int skip, int limit
new TypeReference<List<ai.rhesis.sdk.entities.Test>>() {});
}

public TestSet update(TestSet testSet) {
if (testSet.id() == null) {
throw new IllegalArgumentException("Cannot update a TestSet without an ID");
}
return httpClient.put("/test_sets/" + testSet.id(), testSet, TestSet.class);
}

public void delete(String id) {
httpClient.delete("/test_sets/" + id);
}
Expand Down
2 changes: 1 addition & 1 deletion src/main/java/ai/rhesis/sdk/entities/TestSet.java
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
import java.util.List;
import lombok.Builder;

@Builder
@Builder(toBuilder = true)
public record TestSet(
@JsonProperty("id") String id,
@JsonProperty("name") String name,
Expand Down
12 changes: 6 additions & 6 deletions src/main/java/ai/rhesis/sdk/synthesizers/ConfigSynthesizer.java
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,11 @@ public TestSet generate(int numTests) {
generatedTests.addAll(generateSingleTurnBatch(renderedPrompt));
}

return new TestSet(
null,
"Synthesized TestSet",
"Generated with ConfigSynthesizer",
TestType.SINGLE_TURN,
generatedTests);
String name = config.getTestSetName() != null ? config.getTestSetName() : "Synthesized TestSet";
String description =
config.getTestSetDescription() != null
? config.getTestSetDescription()
: "Generated with ConfigSynthesizer";
return new TestSet(null, name, description, TestType.SINGLE_TURN, generatedTests);
}
}
12 changes: 6 additions & 6 deletions src/main/java/ai/rhesis/sdk/synthesizers/ContextSynthesizer.java
Original file line number Diff line number Diff line change
Expand Up @@ -66,11 +66,11 @@ public TestSet generate(int numTests) {
generatedTests.addAll(generateSingleTurnBatch(renderedPrompt));
}

return new TestSet(
null,
"Synthesized TestSet",
"Generated with ContextSynthesizer",
TestType.SINGLE_TURN,
generatedTests);
String name = config.getTestSetName() != null ? config.getTestSetName() : "Synthesized TestSet";
String description =
config.getTestSetDescription() != null
? config.getTestSetDescription()
: "Generated with ContextSynthesizer";
return new TestSet(null, name, description, TestType.SINGLE_TURN, generatedTests);
}
}
24 changes: 24 additions & 0 deletions src/main/java/ai/rhesis/sdk/synthesizers/GenerationConfig.java
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,17 @@ public class GenerationConfig {
private final List<String> categories;
private final List<String> topics;
private final String additionalContext;
private final String testSetName;
private final String testSetDescription;

private GenerationConfig(Builder builder) {
this.generationPrompt = builder.generationPrompt;
this.requirements = builder.requirements != null ? builder.requirements : new ArrayList<>();
this.categories = builder.categories != null ? builder.categories : new ArrayList<>();
this.topics = builder.topics != null ? builder.topics : new ArrayList<>();
this.additionalContext = builder.additionalContext;
this.testSetName = builder.testSetName;
this.testSetDescription = builder.testSetDescription;
}

public String getGenerationPrompt() {
Expand All @@ -38,6 +42,14 @@ public String getAdditionalContext() {
return additionalContext;
}

public String getTestSetName() {
return testSetName;
}

public String getTestSetDescription() {
return testSetDescription;
}

public static Builder builder() {
return new Builder();
}
Expand All @@ -48,6 +60,8 @@ public static class Builder {
private List<String> categories;
private List<String> topics;
private String additionalContext;
private String testSetName;
private String testSetDescription;

public Builder generationPrompt(String generationPrompt) {
this.generationPrompt = generationPrompt;
Expand All @@ -74,6 +88,16 @@ public Builder additionalContext(String additionalContext) {
return this;
}

public Builder testSetName(String testSetName) {
this.testSetName = testSetName;
return this;
}

public Builder testSetDescription(String testSetDescription) {
this.testSetDescription = testSetDescription;
return this;
}

public GenerationConfig build() {
if (generationPrompt == null) {
throw new IllegalArgumentException("generationPrompt cannot be null");
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,12 +43,15 @@ public TestSet generate(int numTests) {
generatedTests.addAll(generateBatch(currentBatchSize));
}

return new TestSet(
null,
"Synthesized TestSet (Multi-Turn)",
"Generated with MultiTurnSynthesizer based on prompt: " + config.getGenerationPrompt(),
TestType.MULTI_TURN,
generatedTests);
String baseName =
config.getTestSetName() != null ? config.getTestSetName() : "Synthesized TestSet";
String name = baseName + " (Multi-Turn)";
String description =
config.getTestSetDescription() != null
? config.getTestSetDescription()
: "Generated with MultiTurnSynthesizer based on prompt: "
+ config.getGenerationPrompt();
return new TestSet(null, name, description, TestType.MULTI_TURN, generatedTests);
}

private List<Test> generateBatch(int currentBatchSize) {
Expand Down
12 changes: 6 additions & 6 deletions src/main/java/ai/rhesis/sdk/synthesizers/PromptSynthesizer.java
Original file line number Diff line number Diff line change
Expand Up @@ -57,11 +57,11 @@ public TestSet generate(int numTests) {
generatedTests.addAll(generateSingleTurnBatch(renderedPrompt));
}

return new TestSet(
null,
"Synthesized TestSet",
"Generated with PromptSynthesizer based on prompt: " + config.getGenerationPrompt(),
TestType.SINGLE_TURN,
generatedTests);
String name = config.getTestSetName() != null ? config.getTestSetName() : "Synthesized TestSet";
String description =
config.getTestSetDescription() != null
? config.getTestSetDescription()
: "Generated with PromptSynthesizer based on prompt: " + config.getGenerationPrompt();
return new TestSet(null, name, description, TestType.SINGLE_TURN, generatedTests);
}
}
12 changes: 6 additions & 6 deletions src/main/java/ai/rhesis/sdk/synthesizers/Synthesizer.java
Original file line number Diff line number Diff line change
Expand Up @@ -53,12 +53,12 @@ public TestSet generate(int numTests) {
generatedTests.addAll(generateSingleTurnBatch(renderedPrompt));
}

return new TestSet(
null,
"Synthesized TestSet",
"Generated with Synthesizer based on prompt: " + config.getGenerationPrompt(),
TestType.SINGLE_TURN,
generatedTests);
String name = config.getTestSetName() != null ? config.getTestSetName() : "Synthesized TestSet";
String description =
config.getTestSetDescription() != null
? config.getTestSetDescription()
: "Generated with Synthesizer based on prompt: " + config.getGenerationPrompt();
return new TestSet(null, name, description, TestType.SINGLE_TURN, generatedTests);
}

public String getRenderedPrompt() {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ public static void main(String[] args) {
GenerationConfig.builder()
.generationPrompt(
"You are a helpful travel assistant. You must never provide medical advice.")
.testSetName("Travel Assistant Safety Tests")
.requirements(Arrays.asList("Refuses medical advice", "Provides travel itineraries"))
.categories(Arrays.asList("Safety", "Functionality"))
.topics(Arrays.asList("Medical", "Travel"))
Expand All @@ -37,5 +38,10 @@ public static void main(String[] args) {
TestSet pushedTestSet = client.testSets().create(generatedTestSet);

System.out.println("Successfully pushed TestSet! ID: " + pushedTestSet.id());

// Rename the test set after creation
TestSet renamed = pushedTestSet.toBuilder().name("Travel Assistant Safety Tests v2").build();
TestSet updated = client.testSets().update(renamed);
System.out.println("Renamed TestSet to: " + updated.name());
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ public static void main(String[] args) throws Exception {
GenerationConfig.builder()
.generationPrompt(
"You are testing an HR document processor. Generate tests involving reading policy documents.")
.testSetName("HR Document Processor Tests")
.requirements(
Arrays.asList("Accurately summarizes policies", "Identifies vacation days"))
.categories(Arrays.asList("Functionality", "Document Processing"))
Expand Down
39 changes: 29 additions & 10 deletions src/test/java/ai/rhesis/sdk/examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,20 +80,20 @@ Replace the class name with any example listed below.
| Example | Description |
|---------|-------------|
| `ExecuteTestSetExample` | Trigger a test set run against an endpoint — parallel mode, sequential mode, and with custom metrics. |
| `TestRunWorkflowExample` | Full lifecycle: list runs, inspect results, fetch stats, get last run, and rescore. |
| `TestRunWorkflowExample` | Full lifecycle: list runs, inspect results, get insights, get last run, and rescore. |

### Test Set Management

| Example | Description |
|---------|-------------|
| `TestSetMetricsExample` | List, add, and remove metrics on a test set. Associate and disassociate tests. |

### Analytics & Stats
### Insights

| Example | Description |
|---------|-------------|
| `TestRunStatsExample` | Test run analytics: overall summary, status distribution, most-run test sets, timeline, filtering by mode/months/run IDs. |
| `TestResultStatsExample` | Test result analytics: pass rates by metric, requirement, category, and topic. Timeline trends, per-run summaries, and filtered queries. |
| `TestRunStatsExample` | Test run insights: counts by status, date range filtering with months and start/end dates. |
| `TestResultStatsExample` | Test result insights: pass rates by requirement, category, and topic. Date range queries and failed test result ID retrieval. |

## Quick Reference

Expand All @@ -103,17 +103,36 @@ RhesisClient client = RhesisClient.builder()
.apiKey(System.getenv("RHESIS_API_KEY"))
.build();

// Generate a named test set
GenerationConfig config = GenerationConfig.builder()
.generationPrompt("Test a customer support chatbot")
.testSetName("Support Bot Safety Tests")
.testSetDescription("Adversarial tests for the support chatbot")
.requirements(List.of("Refuses harmful requests", "Stays on topic"))
.build();
TestSet testSet = new MultiTurnSynthesizer(config).generate(10);
client.testSets().create(testSet);

// Rename an existing test set
TestSet existing = client.testSets().get(testSetId);
TestSet renamed = existing.toBuilder().name("New Name").build();
client.testSets().update(renamed);

// Execute a test set
Map<String, Object> result = client.testSets()
.execute(testSetId, endpointId);

// Get test run stats
TestRunStats stats = client.testRuns().stats();
System.out.println("Pass rate: " + stats.overallSummary().passRate() + "%");
// Get insights (replaces stats)
InsightsResponse insights = client.insights()
.get("test_result", List.of("requirement"), List.of("count", "pass_rate"), null);

// Get test result stats by requirement
TestResultStats requirementStats = client.testResults()
.stats(TestResultStatsMode.REQUIREMENT);
// Get insights with date range
InsightsQuery query = InsightsQuery.builder("test_result")
.groupBy(List.of("category"))
.measures(List.of("count"))
.months(6)
.build();
InsightsResponse recent = client.insights().get(query);

// Get last completed run
TestRun lastRun = client.testSets()
Expand Down
Loading
Loading