Service Instance Registry Service

Fit in NEANIAS Ecosystem

The NEANIAS Service Instance Registry provides a list of all NEANIAS services with information regarding their location and health status. NEANIAS Services will be registered dynamically, creating list of all available services and their instances. Service Discovery Integration is performed using the respective API available allowing for consistnet integration at bootstrap and runtime.

The NEANIAS Service Instance Registry Service is backed by a distributed, highly available system. The storage can be replicated across multiple nodes of the service to guarantee data redundancy.

The Service operates as a key / value store. Data will is stored using a unique hierarchical key that allows nesting and can be accessed using the same key.

Restricted access is offered (ACLs) in order to restrict the actions (read/write) clients can perform.

The main interface to NEANIAS Service Instance Registry is offred through a set of low level binding clients. All requests will require authentication and client service will be able to perform updates to specific keys.

Technology

The Service Instance Registry is backed by Apache Zookeer (https://zookeeper.apache.org/).

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.

Distributed System

The service itself is distributed and highly reliable. It handles internally aspects pertinent to its distributed nature such as

  • Consensus
  • Group management
  • Presence protocols

ZooKeeper allows distributed processes to coordinate with each other through a shared hierarchical name space of data registers (znodes), much like a file system. Unlike normal file systems ZooKeeper provides its clients with high throughput, low latency, highly available, strictly ordered access to the znodes.

  • The performance aspects of ZooKeeper allow it to be used in large distributed systems.
  • The reliability aspects prevent it from becoming the single point of failure in big systems.
  • Its strict ordering allows sophisticated synchronization primitives to be implemented at the client.

The service itself is replicated over a set of machines that comprise the service. These machines maintain an in-memory image of the data tree along with a transaction logs and snapshots in a persistent store.

Hierarchical Model

The namespace provided by ZooKeeper is much like that of a standard file system. A name is a sequence of path elements separated by a slash (“/”). Every znode in ZooKeeper’s name space is identified by a path. And every znode has a parent whose path is a prefix of the znode with one less element; the exception to this rule is root (“/”) which has no parent. Also, exactly like standard file systems, a znode cannot be deleted if it has any children.

The main differences between ZooKeeper and standard file systems are that every znode can have data associated with it (every file can also be a directory and vice-versa) and znodes are limited to the amount of data that they can have. ZooKeeper was designed to store coordination data: status information, configuration, location information, etc. This kind of meta-information is usually measured in kilobytes, if not bytes. ZooKeeper has a built-in sanity check of 1M, to prevent it from being used as a large data store, but in general it is used to store much smaller pieces of data.

The configured path that a service is registered to follows the concept : /registry/application_name/uuid/data where :

  1. application_name is the configured name of the service.
  2. uuid is a randomly generated uuid indicating the running instance.
  3. data is the pre-configured data of the NeaniasServiceInstance class in a json format

So for example the below data :

{
	"name": "service-1",
	"id": "e0cf1144-eb1b-4248-a243-c5b440a63735",
	"url":	"http://neanias.eu/service-1",
	"status": "UP",
	"metadata": {
		"key1": "value1",
		"key2": "value2"
	}
}

will be found under the path /registry/service-1/e0cf1144-eb1b-4248-a243-c5b440a63735

Bindings

The servers that make up the ZooKeeper service must all know about each other. As long as a majority of the servers are available, the ZooKeeper service will be available. Clients must also know the list of servers. The clients create a handle to the ZooKeeper service using this list of servers.

Clients only connect to a single ZooKeeper server. The client maintains a TCP connection through which it sends requests, gets responses, gets watch events, and sends heartbeats. If the TCP connection to the server breaks, the client will connect to a different server. When a client first connects to the ZooKeeper service, the first ZooKeeper server will setup a session for the client. If the client needs to connect to another server, this session will get reestablished with the new server.

Available client bindings exist and are available for the following languages:

  • Java
  • Python
  • C#
  • Node.js
  • C
  • Go
  • Perl
  • Scala
  • Twisted/Python
  • Erlang
  • Haskell
  • Ruby
  • Lua

Examples

The following example showcases a simple integration of the Service Instance Registry to a Spring Boot Java application retrieving available instances. For a ready-to-go sample navigate under the spring-boot-client folder at the https://gitlab.neanias.eu/instance-registry-service/docs repository.

These example aim to provide an overview of the integration and do not reflect the final hierarchy model or security considerations. ZooKeeper resources provide a lot of information on most of the common and exotic topics that may need to be considered.

Configuring the dependencies :

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.3.4.RELEASE</version>
        <relativePath/>
    </parent>

    <properties>
        <java.version>11</java.version>

        <curator.version>5.1.0</curator.version>
        <zookeeper.version>3.6.2</zookeeper.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.apache.curator</groupId>
            <artifactId>curator-recipes</artifactId>
            <version>${curator.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.curator</groupId>
            <artifactId>curator-x-discovery</artifactId>
            <version>${curator.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.curator</groupId>
            <artifactId>curator-framework</artifactId>
            <version>${curator.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.zookeeper</groupId>
            <artifactId>zookeeper</artifactId>
            <version>${zookeeper.version}</version>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
            <exclusions>
                <exclusion>
                    <groupId>org.junit.vintage</groupId>
                    <artifactId>junit-vintage-engine</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
    </dependencies>

Configuring the application properties :

spring.application.name=service-1
spring.application.url=http://neanias.eu/service-1

zookeeper.connect-string=localhost:2281

server.port=8092
server.ssl.enabled=true
server.ssl.protocol=TLSv1.2
server.ssl.key-store=classpath:ssl/client-keystore.jks
server.ssl.key-store-password=password
server.ssl.key-store-type=JKS
server.ssl.trust-store=classpath:ssl/client-trustore.jks
server.ssl.trust-store-password=password
server.ssl.trust-store-type=JKS

zookeeper.ssl.client.certificate.alias=client

Creating a model containing the necessary registry information :

  • The name property indicates the name of the service.
  • The serviceId property indicates a uuid of the current running service instance as many instances of the same service can be up and running.
  • The url property indicates the url of the service.
  • The status property is configured to have the UP, DOWN, NOT_AVAILABLE values. Upon startup, the service registers with status UP and upon termination it deregisters with status DOWN.
  • The metadata property indicates extra information that the service can have.
package eu.neanias.registry.demo.registry;

import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.annotation.JsonInclude;
import com.fasterxml.jackson.annotation.JsonProperty;
import org.apache.curator.x.discovery.ServiceInstance;
import org.apache.curator.x.discovery.ServiceType;

import java.time.Instant;
import java.util.Map;

@JsonIgnoreProperties({"address", "enabled", "payload", "port", "secure", "sslPort", "serviceType", "uriSpec"})
@JsonInclude(JsonInclude.Include.NON_EMPTY)
public class NeaniasServiceInstance extends ServiceInstance<Object> {
	private String url;
	private ServiceStatus status;
	private Map<String, String> metadata;

	@JsonCreator
	public NeaniasServiceInstance(
		@JsonProperty("name") String serviceName,
		@JsonProperty("id") String serviceId,
		@JsonProperty("url") String url,
		@JsonProperty("status") ServiceStatus status,
		@JsonProperty("metadata") Map<String, String> metadata
	) {
		super(serviceName, serviceId, null, null, null, null, Instant.now().toEpochMilli(), ServiceType.DYNAMIC, null);
		this.url = url;
		this.status = status;
		this.metadata = metadata;
	}

	public String getUrl() {
		return url;
	}

	public void setUrl(String url) {
		this.url = url;
	}

	public ServiceStatus getStatus() {
		return status;
	}

	public void setStatus(ServiceStatus status) {
		this.status = status;
	}

	public Map<String, String> getMetadata() {
		return metadata;
	}

	public void setMetadata(Map<String, String> metadata) {
		this.metadata = metadata;
	}

	public enum ServiceStatus {
		UP, DOWN, NOT_AVAILABLE
	}
}

Configuring an object serializer :

package eu.neanias.registry.demo.registry;

import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.curator.x.discovery.ServiceInstance;
import org.apache.curator.x.discovery.details.InstanceSerializer;

public class NeaniasJsonInstanceSerializer implements InstanceSerializer<Object> {
	private final ObjectMapper mapper = new ObjectMapper();

	public byte[] serialize(ServiceInstance<Object> instance) throws Exception {
		return this.mapper.writeValueAsBytes(instance);
	}

	public ServiceInstance<Object> deserialize(byte[] bytes) throws Exception {
		return this.mapper.readValue(bytes, NeaniasServiceInstance.class);
	}
}

Configuring a custom registry implementation :

package eu.neanias.registry.demo.registry;

public interface NeaniasInstanceRegistryService {
	void register(NeaniasServiceInstance serviceInstance);

	void deregister(NeaniasServiceInstance serviceInstance);

	void updateRegistration(NeaniasServiceInstance serviceInstance);

	void close();
}
package eu.neanias.registry.demo.registry;

import org.apache.curator.x.discovery.ServiceDiscovery;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import org.springframework.util.ReflectionUtils;

import java.io.IOException;

@Component
public class DefaultNeaniasInstanceRegistryService implements NeaniasInstanceRegistryService {
	private final ServiceDiscovery<Object> serviceDiscovery;

	@Autowired
	public DefaultNeaniasInstanceRegistryService(ServiceDiscovery<Object> serviceDiscovery) {
		this.serviceDiscovery = serviceDiscovery;
	}

	private ServiceDiscovery<Object> getServiceDiscovery() {
		return this.serviceDiscovery;
	}

	@Override
	public void register(NeaniasServiceInstance serviceInstance) {
		try {
			this.getServiceDiscovery().registerService(serviceInstance);
		} catch (Exception e) {
			ReflectionUtils.rethrowRuntimeException(e);
		}
	}

	@Override
	public void deregister(NeaniasServiceInstance serviceInstance) {
		try {
			this.getServiceDiscovery().unregisterService(serviceInstance);
		} catch (Exception e) {
			ReflectionUtils.rethrowRuntimeException(e);
		}
	}

	public void afterSingletonsInstantiated() {
		try {
			this.getServiceDiscovery().start();
		} catch (Exception e) {
			ReflectionUtils.rethrowRuntimeException(e);
		}
	}

	@Override
	public void updateRegistration(NeaniasServiceInstance serviceInstance) {
		try {
			this.getServiceDiscovery().updateService(serviceInstance);
		} catch (Exception e) {
			ReflectionUtils.rethrowRuntimeException(e);
		}
	}

	@Override
	public void close() {
		try {
			this.getServiceDiscovery().close();
		} catch (IOException e) {
			ReflectionUtils.rethrowRuntimeException(e);
		}
	}
}

Configuring a custom service discovery implementation for all services - including the service’s possible many instances - registered with status UP :

package eu.neanias.registry.demo.discovery;

import eu.neanias.registry.demo.registry.NeaniasServiceInstance;

import java.util.List;

public interface NeaniasDiscoveryService {
	public List<NeaniasServiceInstance> getInstances(String name) throws Exception;
}
package eu.neanias.registry.demo.discovery;

import eu.neanias.registry.demo.registry.NeaniasServiceInstance;
import org.apache.curator.x.discovery.ServiceDiscovery;
import org.apache.curator.x.discovery.ServiceInstance;
import org.springframework.stereotype.Component;

import java.util.Collection;
import java.util.List;
import java.util.stream.Collectors;

@Component
public class DefaultNeaniasDiscoveryService implements NeaniasDiscoveryService {
	private final ServiceDiscovery<Object> discoveryService;

	public DefaultNeaniasDiscoveryService(ServiceDiscovery<Object> discoveryService) {
		this.discoveryService = discoveryService;
	}

	private ServiceDiscovery<Object> getServiceDiscovery() {
		return this.discoveryService;
	}

	@Override
	public List<NeaniasServiceInstance> getInstances(String name) throws Exception {
		Collection<ServiceInstance<Object>> serviceInstances = getServiceDiscovery().queryForInstances(name);
		return serviceInstances.stream().map(instance -> (NeaniasServiceInstance) instance).filter(instance -> NeaniasServiceInstance.ServiceStatus.UP.equals(instance.getStatus())).collect(Collectors.toList());
	}
}

Configuring Curator client and establishing a connection :

package eu.neanias.registry.demo.configuration;

import eu.neanias.registry.demo.certificate.CertificateLoader;
import eu.neanias.registry.demo.registry.NeaniasJsonInstanceSerializer;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.framework.api.ACLProvider;
import org.apache.curator.retry.ExponentialBackoffRetry;
import org.apache.curator.x.discovery.ServiceDiscovery;
import org.apache.curator.x.discovery.details.InstanceSerializer;
import org.apache.curator.x.discovery.details.ServiceDiscoveryImpl;
import org.apache.zookeeper.ZooDefs;
import org.apache.zookeeper.data.ACL;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.security.KeyStoreException;
import java.security.cert.CertificateEncodingException;
import java.security.cert.X509Certificate;
import java.util.ArrayList;
import java.util.List;

@Configuration
public class CuratorConfiguration {
	@Bean
	public ServiceDiscovery<Object> serviceDiscovery(CuratorFramework curatorFramework) {
		return new ServiceDiscoveryImpl<>(curatorFramework, "registry", instanceSerializer(), null, true);
	}

	@Bean
	public CuratorFramework curatorFramework(
		CertificateLoader certificateLoader,
		@Value("${zookeeper.connect-string}") String connectString,
		@Value("${zookeeper.ssl.client.certificate.alias}") String alias
	) throws KeyStoreException, CertificateEncodingException {

		X509Certificate certificate = (X509Certificate) certificateLoader.loadCertificate(alias);
		byte[] certificateData = certificate.getEncoded();

		CuratorFramework curator =
			CuratorFrameworkFactory.builder()
				.connectString(connectString)
				.retryPolicy(new ExponentialBackoffRetry(2000, 5))
				.authorization("x509", certificateData)
				.aclProvider(aclProvider())
				.build();

		curator.start();

		return curator;
	}

	private ACLProvider aclProvider() {
		return new ACLProvider() {
			@Override
			public List<ACL> getDefaultAcl() {
				List<ACL> acls = new ArrayList<>();
				ACL restrictedPermission = new ACL(ZooDefs.Perms.ALL, ZooDefs.Ids.AUTH_IDS);
				ACL defaultPermission = new ACL(ZooDefs.Perms.READ, ZooDefs.Ids.ANYONE_ID_UNSAFE);
				acls.add(restrictedPermission);
				acls.add(defaultPermission);
				return acls;
			}

			@Override
			public List<ACL> getAclForPath(String path) {
				List<ACL> acls = new ArrayList<>();
				ACL restrictedPermission = new ACL(ZooDefs.Perms.ALL, ZooDefs.Ids.AUTH_IDS);
				ACL defaultPermission = new ACL(ZooDefs.Perms.READ, ZooDefs.Ids.ANYONE_ID_UNSAFE);
				acls.add(restrictedPermission);
				acls.add(defaultPermission);
				return acls;
			}
		};
	}

	@Bean
	public InstanceSerializer<Object> instanceSerializer() {
		return new NeaniasJsonInstanceSerializer();
	}
}

Configuring a loader for the client’s certificate :

package eu.neanias.registry.demo.certificate;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.core.env.Environment;
import org.springframework.core.io.ResourceLoader;
import org.springframework.stereotype.Component;

import java.io.FileInputStream;
import java.io.IOException;
import java.security.KeyStore;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.cert.Certificate;
import java.security.cert.CertificateException;

@Component
public class CertificateLoader {
    private static final String DEFAULT_KEY_STORE_TYPE = "JKS";

    private final KeyStore keyStore;

    @Autowired
    public CertificateLoader(Environment environment, ResourceLoader resourceLoader) throws KeyStoreException, IOException, CertificateException, NoSuchAlgorithmException {
        String keyStoreFile = environment.getProperty("server.ssl.key-store");
        String keyStorePassword = environment.getProperty("server.ssl.key-store-password");
        String keyStoreType = environment.getProperty("server.ssl.key-store-type");

        if (keyStoreFile == null || keyStoreFile.isBlank()) {
            throw new IllegalArgumentException("Property server.ssl.key-store must be defined");
        }
        if (keyStorePassword == null || keyStorePassword.isBlank()) {
            throw new IllegalArgumentException("Property server.ssl.key-store-password must be defined");
        }
        if (keyStoreType == null || keyStoreType.isBlank()) {
            keyStoreType = DEFAULT_KEY_STORE_TYPE;
        }

        keyStore = KeyStore.getInstance(keyStoreType);

        FileInputStream keyStoreFileInput = new FileInputStream(resourceLoader.getResource(keyStoreFile).getFile());
        keyStore.load(keyStoreFileInput, keyStorePassword.toCharArray());
    }

    public Certificate loadCertificate(String alias) throws KeyStoreException {
        return keyStore.getCertificate(alias);
    }
}

Usage of the registration and discovery implementations :

package eu.neanias.registry.demo;

import eu.neanias.registry.demo.registry.NeaniasServiceInstance;
import eu.neanias.registry.demo.registry.NeaniasInstanceRegistryService;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.event.ContextClosedEvent;
import org.springframework.context.event.ContextRefreshedEvent;
import org.springframework.context.event.EventListener;
import org.springframework.stereotype.Component;

import java.util.Map;
import java.util.UUID;

@Component
public class InstanceRegistrationHandler {
	private static final Logger logger = LoggerFactory.getLogger(InstanceRegistrationHandler.class);

	private final NeaniasInstanceRegistryService registry;

	private final String serviceName;
	private final String serviceInstanceId;
	private final String serviceUrl;

	@Autowired
	public InstanceRegistrationHandler(NeaniasInstanceRegistryService registry, @Value("${spring.application.name}") String serviceName, @Value("${spring.application.url}") String serviceUrl) {
		this.registry = registry;
		this.serviceName = serviceName;
		this.serviceInstanceId = UUID.randomUUID().toString();
		this.serviceUrl = serviceUrl;
	}

	@EventListener
	public void onApplicationEvent(ContextRefreshedEvent event) throws Exception {
		Map<String, String> metadata = Map.of(
			"key1", "value1",
			"key2", "value2"
		);

		NeaniasServiceInstance serviceInstance = new NeaniasServiceInstance(serviceName, serviceInstanceId, serviceUrl, NeaniasServiceInstance.ServiceStatus.UP, metadata);
		registry.register(serviceInstance);
		logger.info("Registered service {} with instance id {}", serviceName, serviceInstanceId);
	}

	@EventListener
	public void onApplicationEvent(ContextClosedEvent event) {
		NeaniasServiceInstance serviceInstance = new NeaniasServiceInstance(serviceName, serviceInstanceId, serviceUrl, NeaniasServiceInstance.ServiceStatus.DOWN, null);
		registry.deregister(serviceInstance);
		logger.info("Deregistered service {} with instance id {}", serviceName, serviceInstanceId);
	}
}

Configuring a test suite for the above implementation :

package eu.neanias.registry.demo;

import eu.neanias.registry.demo.discovery.NeaniasDiscoveryService;
import eu.neanias.registry.demo.registry.NeaniasServiceInstance;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.test.context.SpringBootTest;

import java.util.List;
import java.util.stream.Collectors;

import static org.assertj.core.api.Assertions.assertThat;

@SpringBootTest
public class NeaniasInstanceRegistryServiceTests {
	static {
		System.setProperty("zookeeper.clientCnxnSocket", "org.apache.zookeeper.ClientCnxnSocketNetty");
		System.setProperty("zookeeper.client.secure", "true");
		System.setProperty("zookeeper.ssl.client.enable", "true");
		System.setProperty("zookeeper.ssl.keyStore.location", "{{your_path}}\\src\\main\\resources\\ssl\\client1.keystore");
		System.setProperty("zookeeper.ssl.keyStore.password", "password");
		System.setProperty("zookeeper.ssl.keyStore.type", "JKS");
		System.setProperty("zookeeper.ssl.trustStore.location", "{{your_path}}\\src\\main\\resources\\ssl\\client1.truststore");
		System.setProperty("zookeeper.ssl.trustStore.password", "password");
		System.setProperty("zookeeper.ssl.trustStore.type", "JKS");
		System.setProperty("zookeeper.ssl.protocol", "TLSv1.2");
	}

	@Autowired
	private NeaniasDiscoveryService discoveryService;

	@Value("${spring.application.name}")
	private String serviceName;

	@Value("${spring.application.url}")
	private String serviceUrl;

	@Test
	public void testDiscoveryService() throws Exception {
		List<String> discoveredUrls = discoveryService.getInstances(serviceName).stream().map(NeaniasServiceInstance::getUrl).collect(Collectors.toList());
		assertThat(discoveredUrls).contains(serviceUrl);
	}
}

Permissions and access control configuration

The current implementation grants specific access permissions to every authenticated service. A service that registers itself needs to have all the access permissions granted so that it can manage it’s registration accordingly. It must also be visible and discoverable by every other service, leading to granting the rest of the services with only the READ permission.This behaviour can be overriden by changing the getDefaultAcl and getAclForPath methods of the aclProvider in the CuratorConfiguration class.

Client configuration

To establish an ssl connection, the client must create a keystore file containing an entry with the client’s private/public key pair, and a truststore file containing a public trusted certificate.

Keystore configuration

In order to create a keystore using a private key (ex. client.pem) and a public certificate (ex. client.cert), create a keystore in pkcs12 format and then transform it into a jks format :

1. openssl pkcs12 -export -in client.cert -inkey client.pem -out clientkeystore.p12
2. keytool -importkeystore -srckeystore clientkeystore.p12 -srcstoretype pkcs12 -destkeystore client.jks -deststoretype JKS

The created client.jks keystore contains an entry combining the information from both private and public keys but has the default alias, so in order to change it into the correct alias name (ex. client-name) :

1. keytool -changealias -alias default-alias -destalias client-name -keystore client.jks

Truststore configuration

In order to create a truststore using a trusted public certificate from a server (ex. server.cert) with an alias (ex. server-name) :

1. keytool -import -alias server-name -file server.cert -storetype JKS -keystore client-truststore.jks

Java VM options

-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
-Dzookeeper.client.secure=true
-Dzookeeper.ssl.keyStore.location=path\client.jks
-Dzookeeper.ssl.keyStore.password=password
-Dzookeeper.ssl.keyStore.type=JKS
-Dzookeeper.ssl.trustStore.location=path\client-truststore.jks
-Dzookeeper.ssl.trustStore.password=password
-Dzookeeper.ssl.trustStore.type=JKS