| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-03 | |||
| 16:56:26 | sdague | and there are as many successes as fails on mriedem's issue | |
| 16:56:32 | clarkb | the format issue is purely a unicode problem? | |
| 16:56:40 | sdague | clarkb: that is my believe | |
| 16:56:44 | sdague | belief | |
| 16:56:58 | melwitt | sdague: oh, so something else about the change to the journal logger has just made this happen or timed in a way that we see it in gate failures now? | |
| 16:57:04 | sdague | I'm actually starting to think it's a red herring | |
| 16:57:27 | sdague | melwitt: I expect the deadlocking that was causing dansmith's bugs is actually the issue | |
| 16:58:23 | melwitt | sdague: yeah, I think you're right | |
| 16:58:30 | sdague | https://review.openstack.org/#/c/462163/2/functions will fix that | |
| 16:58:56 | sdague | melwitt: that being said, I still think there is a general utf8 bug in ovo repr that would be good to fix :) | |
| 16:59:13 | sdague | at least adding utf8 tests to ovo | |
| 16:59:20 | cfriesen | where does the service version get determined? I don't see it in the Service object. | |
| 17:00:48 | clarkb | sdague: so with that change in place journald will still get the service logs out of stdout? we just won't use the native writer right? | |
| 17:00:57 | sdague | clarkb: correct | |
| 17:01:12 | sdague | systemd will still injest them | |
| 17:01:44 | sdague | they just miss out on the native annotations, which include things like code line, correct process_id, and extras | |
| 17:01:47 | melwitt | sdague: yeah ... I'll look at that. it's not straightforward to me bc the __repr__ intentionally converts to bytes (opposite of what we want for the logging) and it seems to work fine outputting to stdout for example, so I don't yet understand what should be done there | |
| 17:02:27 | sdague | melwitt: yeh, I'm not sure I do either. But it also doesn't look like any of the ovo tests have utf8 strings in them | |
| 17:02:33 | sdague | that would at least be a start | |
| 17:04:33 | melwitt | sdague: ack. will do | |
| 17:07:35 | mriedem | cfriesen: nova.service._update_service_ref | |
| 17:08:56 | mriedem | cfriesen: and __init__ of the Service object | |
| 17:08:58 | mriedem | when creating a new one | |
| 17:09:02 | mriedem | self.version = SERVICE_VERSION | |
| 17:10:17 | sdague | mriedem: what things do you want in syslog.txt ? | |
| 17:24:13 | mriedem | sdague: not the openstack service logs | |
| 17:25:57 | sdague | mriedem: sure, but we have to build back up which things you want in there | |
| 17:28:03 | sdague | https://review.openstack.org/#/c/462231/ | |
| 17:29:10 | mriedem | sdague: i don't know honestly, syslog was my place to go look for native process things on the system, like libvirt, mysql, tgtd, kernel stuff, etc | |
| 17:29:16 | mriedem | i'd use the libvirt logs for libvirt debug info, | |
| 17:29:22 | mriedem | and the openstack service logs for openstack service things | |
| 17:29:55 | mriedem | the journal gz files are less useful to me since i can't use them in the browser, and have to pull them down | |
| 17:30:32 | sdague | mriedem: right, well, we're in systemd world, everything is in the journal | |
| 17:30:46 | sdague | and the export has to be based on inclusion, not exclusion | |
| 17:31:12 | sdague | so going to start with kernel / sudo | |
| 17:31:20 | sdague | and if you want other topics please propose | |
| 17:32:28 | openstackgerrit | Gábor Antal proposed openstack/nova master: Transform aggregate.remove_host notification https://review.openstack.org/396211 | |
| 17:33:27 | mriedem | sdague: that's likely a fine start, and if i really need to debug anything that i'd normally find in syslog, or look in syslog for, i'll have to pull the journal down | |
| 17:33:45 | mriedem | sdague: that means we might lose some things that we were indexing from syslog into logstash before, | |
| 17:33:49 | mriedem | but that's probably a small set, idk | |
| 17:33:54 | mriedem | libvirt things for sure though... | |
| 17:34:14 | sdague | mriedem: it does, so if there are important things, I'd suggest just adding topics | |
| 17:34:43 | sdague | mriedem: there is a reason I'm asking you what topics you want, because I expect you are one of the top users :) | |
| 17:34:55 | sdague | so I'm totally good with mriedems private label topics list | |
| 17:35:02 | clarkb | dnsmasq | |
| 17:35:13 | clarkb | thats been useful to debug dhcp things int he past | |
| 17:35:38 | sdague | clarkb: it has, though usually not with a spot check | |
| 17:35:52 | mriedem | i dumped the current open e-r queries that rely on syslog in the patch | |
| 17:35:54 | sdague | it will still be in the journal dump | |
| 17:36:02 | mriedem | i haven't dug into what they are looking for though, because i'm lunching | |
| 17:36:07 | sdague | mriedem: ok | |
| 17:36:14 | sdague | speaking of, I need to do that | |
| 17:36:26 | sdague | then will be intermittent for the rest of the day | |
| 17:37:02 | sdague | I fast approved the disable on use_journal - https://review.openstack.org/#/c/462163/ - so hopefully in a couple of hours that will be in | |
| 17:38:25 | mriedem | cool | |
| 17:39:25 | mriedem | cfriesen: are you going to be at the summit? | |
| 17:39:46 | mriedem | because there is a session where all of your questions can be answered! | |
| 17:40:06 | mriedem | unless they have something to do with numa and pci, then we'll be asking you for the answeres | |
| 17:40:08 | mriedem | *answers | |
| 17:51:01 | mriedem | jaypipes: i had no idea wookiee had 2 es | |
| 17:51:36 | jaypipes | mriedem: you're welcome :) | |
| 17:51:41 | jaypipes | TIL... | |
| 17:51:42 | mriedem | i actually like this format | |
| 17:51:51 | mriedem | puts mine to shame | |
| 17:51:54 | mriedem | damn you | |
| 17:53:34 | mriedem | rxtx factors, ha | |
| 17:53:49 | mriedem | didn't i see sfinucan trying to remove all of that? | |
| 17:55:22 | mriedem | or maybe that's a joke b/c rxtx factors would normally be an extra spec, but aren't | |
| 17:55:26 | mriedem | for whatever reason | |
| 17:57:13 | melwitt | dansmith: do you know what is the purpose of this stringify that puts extra quotes around a string? https://github.com/openstack/oslo.versionedobjects/blob/master/oslo_versionedobjects/fields.py#L269 | |
| 17:57:31 | sean-k-mooney | mriedem: aren't rx/tx factors more of a qos thing. e.g. sould they still exist now that neutron has a qos api | |
| 17:57:50 | mriedem | sean-k-mooney: they are only used for port qos yeah | |
| 17:58:04 | dansmith | melwitt: it's because that ends up in the repr, right? | |
| 17:58:20 | mriedem | sean-k-mooney: https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L1088 | |
| 17:58:33 | mriedem | sean-k-mooney: that's the only place we ues them for neutron in nova | |
| 17:58:39 | mriedem | but they probably shouldn't be in nova | |
| 17:58:49 | melwitt | dansmith: yeah. I'm trying to wrap my head around this stuff for when a StringField contains a unicode thing. and I was wondering what the adding of the quotes around the StringField value does if it's already a string? | |
| 17:59:00 | mriedem | if you want qos on your port, create the port in neutron and provide it when creating the server | |
| 17:59:10 | melwitt | dansmith: I'm probably dumbly missing something here | |
| 17:59:13 | dansmith | melwitt: it's just for display | |
| 17:59:37 | dansmith | melwitt: so you end up with Instance<name="foo bar"> and not Instance<name=foo bar> | |
| 17:59:38 | sean-k-mooney | mriedem: so they were part of the old port qos support. i think this inclunces the libvirt xml generation somehow if i remember correctly | |
| 17:59:45 | melwitt | dansmith: right, okay. thank you | |
| 18:02:33 | mriedem | sean-k-mooney: i don't see that happening | |
| 18:03:20 | mriedem | sean-k-mooney: in fact the only place i see rxtx* in nova/virt is in the xenapi driver | |
| 18:03:33 | sean-k-mooney | ya im looking at the git repo currently and i cant find it either but i taguht it used too.. | |
| 18:03:36 | mriedem | and it's based on a flag that's only set if you're using nova-network | |
| 18:03:38 | sean-k-mooney | well its set here https://github.com/openstack/nova/blob/501a0fe65afabb7e6a8b1e2d1ca0fbcea941d2d1/nova/tests/unit/virt/libvirt/test_vif.py#L507 | |
| 18:03:45 | sean-k-mooney | but i dont know if its used | |
| 18:03:57 | mriedem | sean-k-mooney: that's just a flavor | |
| 18:04:02 | mriedem | the rxtx_factor is defined on the nova flavor | |
| 18:04:03 | sean-k-mooney | yes | |
| 18:04:20 | mriedem | we don't use that when plugging the vifs in libvirt though, or creating the domain xml, from what i can tell | |
| 18:04:38 | johnthetubaguy | mriedem: thats a rackspace special for quark | |
| 18:04:44 | johnthetubaguy | mriedem: AFAIK | |
| 18:04:48 | mriedem | it's probably time to deprecate rxtx_factor in our flavor api | |
| 18:04:59 | mriedem | and say if you need qos on ports, get it from neutron | |
| 18:05:09 | johnthetubaguy | mriedem: +1 that | |
| 18:05:42 | mriedem | sdague: jaypipes: dansmith: let's remember to put something in the onboarding session about "if you don't know why it's in nova, or what uses/used it, it's probably something to enable out of tree things for rackspace" | |
| 18:05:49 | mriedem | i need constant reminding of that | |