CAT 使用小结

CAT具体的设计思想、实现原理在这我就不罗列了,本文主要是记录一下在使用CAT的过程中遇到的一些问题,比如分布式logview,Cache、DB埋点监控等,问题不多,但是比较典型。

(本文涉及的CAT版本为1.3.6)

1、分布式 logview 的日志树串联实现

目前使用过两种,一种是基于 dubbo 应用的 rpc 调用,一种是基于 http 请求的 rest 服务调用。首先说下 message tree 的实现,追踪跨服务的消息时,通过根消息 id 和父级消息 id 及子消息 id 三个属性进行消息串联,组成消息树。关键点在 tree 的三个 id 的获得和传递。

这里有两点,第一是 CAT 消息树生成原理

我们需要实现 Cat 的 Context 上下文,然后通过 Cat.logRemoteCallClient(context) 生成包含节点数据的上下文对象(方法中通过创建消息树对象来获取各节点的消息 id,填充给上下文),当远程服务端接收到这个 context 时,使用 Cat.logRemoteCallServer(context) 方法,读取各节点消息 id,组建消息树。

第二是消息应如何传递

dubbo 应用的 rpc 调用方式:调用过程要传递的 rpc 上下文,其中包含调用信息、参数以及状态信息等,可以把消息 id 信息放到 RpcContext 中,然后通过调用 Invocation 对象的 invoke 方法,将消息传递至服务端。最后,通过dubbo的 spi 拓展机制,实现 com.alibaba.dubbo.rpc.Filter,用来获取 rpcContext 的内容。

rest 风格的http请求方式:调用时,在服务请求方把消息id信息放到 Http-Header 中,在服务提供方,用filter 拦截,并获得 http-header 中的消息 id,这样通过埋点,串联起消息树。

废话不多说了,上码吧。

1).dubbo 调用方式部分实现(首先要清楚 dubbo 的 spi 相关配置,CAT监控的配置等)

public class DubboCatFilter implements Filter {

    private static final ThreadLocal<Cat.Context> CAT_CONTEXT = new ThreadLocal<Cat.Context>();

    @Override
    public Result invoke(Invoker<?> invoker, Invocation invocation) throws RpcException {
        URL url = invoker.getUrl();
        String sideKey = url.getParameter(Constants.SIDE_KEY);
        String loggerName = invoker.getInterface().getSimpleName() + "." + invocation.getMethodName();
        String type = "PigeonCall";
        if (Constants.PROVIDER_SIDE.equals(sideKey)) {
            type = "PigeonService";
        }
        Transaction t = Cat.newTransaction(type, loggerName);
        Result result = null;
        try {
            Cat.Context context = getContext();
            if (Constants.CONSUMER_SIDE.equals(sideKey)) {
                createConsumerCross(url, t);
                Cat.logRemoteCallClient(context);
            } else {
                createProviderCross(url, t);
                Cat.logRemoteCallServer(context);
            }
            setAttachment(context);
            result = invoker.invoke(invocation);

            if (result.hasException()) {
                //给调用接口出现异常进行打点
                Throwable throwable = result.getException();
                Event event = null;
                if (RpcException.class == throwable.getClass()) {
                    Throwable caseBy = throwable.getCause();
                    if (caseBy != null && caseBy.getClass() == TimeoutException.class) {
                        event = Cat.newEvent("DUBBO_TIMEOUT_ERROR", loggerName);
                    } else {
                        event = Cat.newEvent("DUBBO_REMOTING_ERROR", loggerName);
                    }
                } else if (RemotingException.class.isAssignableFrom(throwable.getClass())) {
                    event = Cat.newEvent("DUBBO_REMOTING_ERROR", loggerName);
                }else{
                    event = Cat.newEvent("DUBBO_BIZ_ERROR", loggerName);
                }
                event.setStatus(result.getException());
                completeEvent(event);
                t.addChild(event);
                t.setStatus(result.getException().getClass().getSimpleName());
            } else {
                t.setStatus(Message.SUCCESS);
            }
            return result;
        } catch (RuntimeException e) {
            Event event = null;
            if (RpcException.class == e.getClass()) {
                Throwable caseBy = e.getCause();
                if (caseBy !=null && caseBy.getClass() == TimeoutException.class) {
                    event = Cat.newEvent("DUBBO_TIMEOUT_ERROR", loggerName);
                } else {
                    event = Cat.newEvent("DUBBO_REMOTING_ERROR", loggerName);
                }
            } else {
                event = Cat.newEvent("DUBBO_BIZ_ERROR", loggerName);
            }
            event.setStatus(e);
            completeEvent(event);
            t.addChild(event);
            t.setStatus(e.getClass().getSimpleName());
            if (result == null) {
                throw e;
            } else {
                return result;
            }
        } finally {
            t.complete();
            CAT_CONTEXT.remove();
        }
    }

    static class DubboCatContext implements Cat.Context {
        private Map<String,String> properties = new HashMap<String, String>();

        @Override
        public void addProperty(String key, String value) {
            properties.put(key,value);
        }

        @Override
        public String getProperty(String key) {
            return properties.get(key);
        }
    }

    private void setAttachment(Cat.Context context) {
        RpcContext.getContext().setAttachment(Cat.Context.ROOT,context.getProperty(Cat.Context.ROOT));
        RpcContext.getContext().setAttachment(Cat.Context.CHILD,context.getProperty(Cat.Context.CHILD));
        RpcContext.getContext().setAttachment(Cat.Context.PARENT,context.getProperty(Cat.Context.PARENT));
    }

    private Cat.Context getContext(){
        Cat.Context context = CAT_CONTEXT.get();
        if (context==null) {
            context = initContext();
            CAT_CONTEXT.set(context);
        }
        return context;
    }

    private Cat.Context initContext() {
        Cat.Context context = new DubboCatContext();
        Map<String,String> attachments = RpcContext.getContext().getAttachments();
        if (attachments!=null&&attachments.size()>0) {
            for (Map.Entry<String,String> entry:attachments.entrySet()) {
                if (Cat.Context.CHILD.equals(entry.getKey()) || Cat.Context.ROOT.equals(entry.getKey()) || Cat.Context.PARENT.equals(entry.getKey())) {
                    context.addProperty(entry.getKey(),entry.getValue());
                }
            }
        }
        return context;
    }

    private void createConsumerCross(URL url, Transaction t) {
        Event crossAppEvent = Cat.newEvent("PigeonCall.app", getProviderAppName(url));
        Event crossServerEvent = Cat.newEvent("PigeonCall.server", url.getHost());
        Event crossPortEvent = Cat.newEvent("PigeonCall.port", url.getPort() + "");
        crossAppEvent.setStatus(Event.SUCCESS);
        crossServerEvent.setStatus(Event.SUCCESS);
        crossPortEvent.setStatus(Event.SUCCESS);
        completeEvent(crossAppEvent);
        completeEvent(crossPortEvent);
        completeEvent(crossServerEvent);
        t.addChild(crossAppEvent);
        t.addChild(crossPortEvent);
        t.addChild(crossServerEvent);
    }

    private void createProviderCross(URL url, Transaction t) {
        String consumerAppName = RpcContext.getContext().getAttachment(Constants.APPLICATION_KEY);
        if (StringUtils.isEmpty(consumerAppName)) {
            consumerAppName = RpcContext.getContext().getRemoteHost() + ":" + RpcContext.getContext().getRemotePort();
        }
        Event crossAppEvent = Cat.newEvent("PigeonService.app", consumerAppName);
        Event crossServerEvent = Cat.newEvent("PigeonService.client", url.getHost());
        crossAppEvent.setStatus(Event.SUCCESS);
        crossServerEvent.setStatus(Event.SUCCESS);
        completeEvent(crossAppEvent);
        completeEvent(crossServerEvent);
        t.addChild(crossAppEvent);
        t.addChild(crossServerEvent);
    }

    private void completeEvent(Event event) {
        AbstractMessage message = (AbstractMessage) event;
        message.setCompleted(true);
    }

}

2).http-restful 调用方式部分实现

CatHttpClientProxy.java

public void requestByGet(String url) {
    Transaction t = Cat.newTransaction("PigeonCall", "method000");
    
    //创建默认的httpClient实例
    CloseableHttpClient httpClient = HttpClients.createDefault();
    RequestConfig requestConfig = RequestConfig.custom()  
                .setConnectTimeout(5000).setConnectionRequestTimeout(1000)  
                .setSocketTimeout(5000).build();
    try {
        HttpGet httpGet = new HttpGet(url);
        httpGet.setConfig(requestConfig);
            
        //串联埋点
        Cat.Context context = new CatHttpContext();
        this.createConsumerCross(url, t);
        Cat.logRemoteCallClient(context);
        httpGet.setHeader(Cat.Context.ROOT, context.getProperty(Cat.Context.ROOT));
        httpGet.setHeader(Cat.Context.PARENT, context.getProperty(Cat.Context.PARENT));
        httpGet.setHeader(Cat.Context.CHILD, context.getProperty(Cat.Context.CHILD));
            
        System.out.println("执行get请求:...." + httpGet.getURI());
        CloseableHttpResponse httpResponse = null;
        //发送get请求
        httpResponse = httpClient.execute(httpGet);//请求返回的Resp,含http的header和执行结果实体Entity
        try {
            //response实体
            HttpEntity entity = httpResponse.getEntity();//不包含header
            if (null != entity) {
                System.out.println("响应状态码:"+ httpResponse.getStatusLine());
                System.out.println("-------------------------------------------------");
                System.out.println("响应内容:" + EntityUtils.toString(entity));
            }
        } finally {
            httpResponse.close();
        }
        t.setStatus(Transaction.SUCCESS);
    } catch (Exception e) {
        e.printStackTrace();
        t.setStatus(e.getClass().getSimpleName());
    } finally {
        t.complete();
        try {
            closeHttpClient(httpClient);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

private void createConsumerCross(String url, Transaction t){
    Event crossAppEvent = Cat.newEvent("PigeonCall.app", "serverName");
    Event crossServerEvent = Cat.newEvent("PigeonCall.server", "serverIp");
    Event crossPortEvent = Cat.newEvent("PigeonCall.port", "serverPort");
    crossAppEvent.setStatus(Event.SUCCESS);
    crossServerEvent.setStatus(Event.SUCCESS);
    crossPortEvent.setStatus(Event.SUCCESS);
    completeEvent(crossAppEvent);
    completeEvent(crossPortEvent);
    completeEvent(crossServerEvent);
    t.addChild(crossAppEvent);
    t.addChild(crossPortEvent);
    t.addChild(crossServerEvent);
}

private void completeEvent(Event event){
    AbstractMessage message = (AbstractMessage) event;
    message.setCompleted(true);
}

private void closeHttpClient(CloseableHttpClient client) throws IOException{
    if (client != null) {
        client.close();
    }
}

2、CAT对 redis 缓存进行详细监控

CAT源码内部对于缓存的识别存在一个 convention 约定,是基于匹配 “Cache.” 字符串的,并且通过判断字符串 “Cache.memcached” 来支持 memcached 监控,可是没有对 redis 做显示支持,需要修改源码,增加判断字符串 “Cache.redis”;

1).修改类:cat-home - com.dianping.cat.report.page.statistics.task.utilization.TransactionReportVisitor.java

增加对 redis 的判断支持:

private static final String REDIS = "Cache.redis";
public TransactionReportVisitor() {
	m_types.add("URL");
	m_types.add("Service");
	m_types.add("PigeonService");
	m_types.add("Call");
	m_types.add("PigeonCall");
	m_types.add("SQL");
	m_types.add(MEMCACHED);
	m_types.add(REDIS);
}
@Override
public void visitType(TransactionType type) {
	String typeName = type.getId();
	Domain domain = m_report.findOrCreateDomain(m_domain);
  
	if ("Service".equals(typeName)) {
		typeName = "PigeonService";
	} else if ("Call".equals(typeName)) {
		typeName = "PigeonCall";
	} else if (typeName.startsWith(MEMCACHED)) {
		typeName = MEMCACHED;
	} else if (typeName.startsWith(REDIS)){
		typeName = REDIS;
	}
	......
}

2).修改类:cat-core - com.dianping.cat.config.server.ServerConfigManager.java

增加对 redis 的判断支持:

public boolean isCacheTransaction(String type) {
	return StringUtils.isNotEmpty(type) && (type.startsWith("Cache.memcached") || type.startsWith("Cache.redis"));
}

3).修改类:cat-consumer - com.dianping.cat.consumer.storage.StorageAnalyzer.java

增加对redis的判断支持:

private void processCacheTransaction(MessageTree tree, Transaction t) {
	String cachePrefix = "Cache.";
	String ip = "Default";
	String domain = tree.getDomain();
	String cacheType = t.getType().substring(cachePrefix.length());
	String name = t.getName();
	String method = name.substring(name.lastIndexOf(":") + 1);
	List<Message> messages = t.getChildren();

	for (Message message : messages) {
		if (message instanceof Event) {
			String type = message.getType();

			if (type.equals("Cache.memcached.server") || type.equals("Cache.redis.server")) {
				ip = message.getName();
				int index = ip.indexOf(":");
				if (index > -1) {
					ip = ip.substring(0, index);
				}
			}
		}
	}
	......
}

3、CAT 对 DB 数据库进行详细监控

如果你的 orm 框架使用的 mybatis,可以考虑通过实现拦截器 Interceptor 来对DB进行底层监控,CAT对数据库的埋点也存在 convention,这里代码中存在 hard code。具体埋点如下:

MappedStatement mappedStatement = (MappedStatement) invocation.getArgs()[0];
//得到类名,方法
String[] strArr = mappedStatement.getId().split("\\.");
String methodName = strArr[strArr.length - 2] + "." + strArr[strArr.length - 1];
Transaction t = Cat.newTransaction("SQL", "methodName");

//获取SQL类型
SqlCommandType sqlCommandType = mappedStatement.getSqlCommandType();
Cat.logEvent("SQL.Method", sqlCommandType.name().toLowerCase());

String JDBC_CONNECTION = "jdbc:mysql://unknown:3306/%s?useUnicode=true";
Cat.logEvent("SQL.Database", String.format(JDBC_CONNECTION, serverIp, dbName));

spring配置如下:

<bean id="sessionFactory" class="org.mybatis.spring.SqlSessionFactoryBean">
    <property name="dataSource" ref="dataSource"/>
    <property name="configLocation" value="classpath:mybatis.xml"/>
    <!-- 插件配置 -->
    <property name="plugins">
        <array>
            <bean class="com.kubbo.java.common.cat.CatMybatisPlugin"></bean>
        </array>
    </property>
</bean>

以上仅罗列了每个问题的一种实现方案,只是给正在研究CAT的同学一个参考思路,个人研究CAT也是刚开始,所说之处不免存在一些纰漏,欢迎指正和交流。

http://my.oschina.net/u/129971/blog/688371

相关推荐