Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-01-20
12:30:56 snikitin I guess I need to add 'is NaN' checks into my code. Did you face with such strange behavior?
12:30:56 snikitin My code can't convert float('NaN') into int and this is the reason of the fails. On my local Xen env everything is ok. I saw 'NaN' values only one time and I don't know how to reproduce it. But on CI I saw normal values only one time. All other times I see 'NaN'.
12:30:56 snikitin BobBall: Hi, Bob! JFYI: I found the reason of diagnostics tempest tests fail. Sometimes (I don't know why) xen returns "NaN" instead of numbers. For example I saw such response in Xen CI job: {"last_update": "1484900660", "cpu0": "NaN", "memory": "NaN"}.
12:32:01 BobBall I suspect this will be because we're getting the stats from the RRDs which are updated every 5 seconds. Perhaps a pause in the tempest test for 10 seconds(?) to ensure some stats are populated would fix it?
12:35:52 snikitin BobBall: Thank you for advice! I will check it. P.S. Even if some stats aren't populated I expect to see something like 'None' or 'null' rather than 'NaN'. strange behavior.
12:38:37 BobBall snikitin: The RRDs are typically based on min, max or averages over a period (as an instant reading doesn't make much sense for things like CPU usage). My guess is that they are providing a number based on the average over X-X seconds - which will be a number divided by 0 as we don't have sufficient time periods. I get what you're saying, but if a 10 second delay fixes the issue I would understand why it is returning "Not a Number"
12:41:54 bauzas sdague: quick question, we deduce the endpoint from the token unless we use the endpoint_override here https://github.com/openstack/keystoneauth/blob/master/keystoneauth1/session.py#L509-L513
12:42:18 bauzas sdague: am I correct if I say that if I'm providing endpoint_override, I'd not lookup the SC ?
12:42:28 bauzas and directly call the placement service
12:42:58 sdague bauzas: looks like... however
12:43:05 sdague if you are really using noauth
12:44:16 sdague then probably you are actually going to want to build a fake client entirely, I think mocking out keystoneauth at all the right points is going to be harder
12:44:57 sean-k-mooney dims: jaypipes looking at https://review.openstack.org/#/c/422838 do we still need to merge this and make a release to fix the python 3.5 support today?
12:45:04 bauzas I don't want to mock out all the keystoneclient
12:45:18 bauzas sdague: just recreating a new Session with an endpoint_override
12:45:33 snikitin BobBall: Ok. I will check it, but I think you right. Anyway I think we should check 'NaN' values in nova, because user can get a 500 error because of it.
12:45:47 sdague bauzas: ok
12:45:57 bauzas anyway, let's try
12:46:21 bauzas just taking a few mins for refueling myself at $lunch, and then back quickly
12:46:26 BobBall snikitin: Sure - the xenapi drive could easily translate NaN to None if it makes more sense up the stack
12:51:59 snikitin BobBall: I checked. It works with sleep(10)! Thank you!
12:53:44 dims sean-k-mooney : nah. no need to rush it. py35 is just experiment anyway at the moment
12:54:09 markus_z sdague: We could make the testing gap for live-migration a little smaller with https://review.openstack.org/#/c/347471/ Would be nice if you could find the time for that.
12:54:26 markus_z mriedem: Maybe you too, I answered your questions there ^
12:55:40 sean-k-mooney dims: ok ill try and get a linux bridge devstack enviornment up and running next week and we can also look into adding a linux bridge job to os-vif also going forward
12:55:53 dims ++ sean-k-mooney thansk!
13:08:26 jaypipes dims: what's the latest status on the mox3 thing? has that revert merged?
13:09:06 dims jaypipes : y, should be green. please rebase if you still see an issue
13:09:18 jaypipes dims: will do. cheers!
13:25:57 mriedem getting this resource providers patch in, which is pretty simple, closes out the aggregates stuff for ocata https://review.openstack.org/#/c/407309/
13:26:01 mriedem needs another core ^
13:26:20 leakypipes mriedem: I'm on it, captain.
13:26:28 leakypipes oh wait, that's mine.
13:26:29 mriedem leakypipes: it's yours
13:26:30 mriedem ha
13:26:30 leakypipes never mind :)
13:27:21 mriedem leakypipes: you could deal with the -1 on this though https://review.openstack.org/#/c/404472/
13:27:43 leakypipes mriedem: ya, will do.
13:28:02 leakypipes mriedem: just trying to get sahid's reserved CPUs patches reviewed.
13:28:08 mriedem melwitt_: sdague: i got the discover_hosts running in d-g http://logs.openstack.org/05/423005/2/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/2704d02/logs/discover-hosts.txt.gz
13:28:50 mriedem needs to be fixed up for stable branches though
13:28:53 mriedem and grenade
13:29:22 mriedem http://logs.openstack.org/05/423005/2/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/d230b29/logs/discover-hosts.txt.gz
13:29:56 openstackgerrit Timofey Durakov proposed openstack/nova: Make all timestamps formats equal https://review.openstack.org/421981
13:32:23 tbachman a little levity
13:32:26 tbachman is
13:32:27 tbachman needed
13:33:33 leakypipes tbachman: indeed :(
13:34:32 mfeoktistov Hello! Dear cores, can you please look at https://review.openstack.org/#/c/312488/
13:36:45 openstackgerrit OpenStack Proposal Bot proposed openstack/nova: Updated from global requirements https://review.openstack.org/422323
13:37:33 sdague mriedem: cool
13:39:52 sdague mriedem: once I get some breakfast and this set of grenade results, I'll take a look
13:40:28 mriedem the d-g change isn't really ready yet, i need to make it conditional on branches, or put a script in devstack to run discover_hosts and then d-g would only run it if that script exists
13:40:37 mriedem https://review.openstack.org/#/c/423005/2/devstack-vm-gate.sh
13:41:23 mriedem if you have input on what's the preferred approach there that would be good
13:42:13 leakypipes tbachman: :)
13:43:45 mdbooth mriedem: NFS https://review.openstack.org/#/c/383859/
13:43:59 mriedem blarg shared file systems
13:44:09 mdbooth I've actually been hammering on this over the last couple of days because it fixes another bug too
13:44:40 mdbooth If you suspend an instance with an attached volume, the volume can be unmounted while it's suspended
13:44:48 mdbooth So it fails when you resume it
13:44:55 mdbooth Confirmed this patch fixes that bug
13:45:53 mdbooth https://review.openstack.org/#/c/383859/
13:46:24 mdbooth Except it's not a race, it's entirely deterministic in this case
13:47:20 mriedem ok
13:47:55 mriedem i'll be honest i'm going to have a very distracted day today, trying to wrap things up before flying out for vacation tomorrow
13:48:03 BobBall snikitin: Glad to hear it
13:48:14 mdbooth Ah, it's already in there
13:49:20 openstackgerrit ChangBo Guo(gcb) proposed openstack/nova: Set access_policy for messaging's dispatcher https://review.openstack.org/408494
13:50:40 mdbooth Unrelated, can anybody check my memory for me: https://review.openstack.org/#/c/376350/ . The behaviour of reboot where is tries to fix potential local hypervisor problems is intentional, right?
13:51:21 mdbooth i.e. recreating backing files and rewriting the domain xml.
13:51:53 mdbooth Both of these behaviours are undesirable, imho, but iiuc they're relied upon by operators. Is that right?
13:52:13 mdbooth sdague: ^^^
13:52:20 openstackgerrit OpenStack Proposal Bot proposed openstack/nova: Updated from global requirements https://review.openstack.org/422323
13:52:25 bauzas mriedem: FYI, the scheduler change is blocked because also another problem due to us not having yet a fixture for our functional tests
13:52:58 bauzas mriedem: sdague: there is a Gabbi fixture in our gabbi branch, but not for the functional tests
13:53:32 bauzas sdague: any reason why when calling the placement API using urllib2 and noauth2 middleware, it still returns 401 ?
13:55:19 mdbooth mriedem: I ran the latest patch version against the devstack nfs CI, btw. It passed.
13:55:24 johnthetubaguy melwitt_: mriedem: I have been reading that quota stuff just now, any reason we can't change ProjectUserQuota to have user_id column as nullable, and just kill the Quota table? (or add a nullable user_id into the Quota table, and kill ProjectUserQuota)?
13:56:05 mriedem johnthetubaguy: i don't know off the top of my head, i know the tables are used based on resource type
13:56:17 mriedem i don't know how much of that is still valid after the move
13:57:49 johnthetubaguy mriedem: no worries, I am going to keep reading
13:58:57 mriedem i would assume things stay the same in those tables when we move to the api db, they might change their function or necessity after the counting quotas stuff is in, but i'd probably lean toward making any changes to the data model in a later release
14:03:10 sahid mdbooth: about https://review.openstack.org/#/c/376350/ - i do not understand your points
14:03:40 sahid if there the domain XML can be caorrupted in someways to we probzbly have to fix the root cause
14:03:50 sahid same for the disks of instance
14:05:46 openstackgerrit OpenStack Proposal Bot proposed openstack/nova: Updated from global requirements https://review.openstack.org/422323
14:05:55 mdbooth sahid: I've posted a hopefully clearer response on the review.
14:05:58 sahid if you can explaining or at least report a bug so i could fix both of these issues but like that it's just you want something broken to stay broken
14:06:11 mdbooth Specific bugs misses the point.
14:07:05 mdbooth sahid: I was also unaware of this apparently undocumented design decision, which is why I'm hoping to get some additional input.
14:07:43 sahid yes because that seems weird the hard-reboot should be helpfull when the guest operating system get stuck
14:08:04 mdbooth It's about the hypervisor rather than the guest OS, I think.
14:08:12 mdbooth Well, I guess it's about either.
14:08:24 sahid but i do not think it's how operators should repair an instance "broken"
14:09:03 mdbooth It's like: steps to take before defenestrating the instance:
14:09:07 mdbooth 1. reboot
14:09:09 mdbooth 2. ...
14:09:37 sahid but we need to be clear on what we are talking about, the qemu process or the guest operating system ?
14:09:57 mdbooth s/qemu process/hypervisor/

Earlier   Later