| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-03 | |||
| 18:34:33 | johnthetubaguy | mriedem: I am guessing there is no way to backport this one, if it misses RC1 right? https://review.openstack.org/#/c/424428 | |
| 18:35:44 | mriedem | johnthetubaguy: not with the object version change, | |
| 18:35:46 | mriedem | but there is rc2 | |
| 18:35:56 | johnthetubaguy | mriedem: what if we just get the object change in? | |
| 18:36:04 | mriedem | heh, so i thought about that yesterday | |
| 18:36:13 | johnthetubaguy | mriedem: but rc2 is a backport I guess | |
| 18:36:22 | mriedem | yes rc2 is a backport to stable/ocata technically | |
| 18:36:35 | mriedem | but it'd also be pre-release | |
| 18:36:36 | superdan | johnthetubaguy: mriedem it's okay if the versions have no gap, and I think this is sufficiently defensive | |
| 18:36:47 | superdan | I mean, it's technically okay if there's no gap | |
| 18:37:20 | johnthetubaguy | superdan: oh, true, the likely hood of moving from stable to a commit without that commit seems fairly tiny | |
| 18:37:46 | superdan | johnthetubaguy: the biggestdeal is if you happen to get conductor on rc1 and compute on rc2 | |
| 18:37:50 | superdan | however, | |
| 18:38:03 | superdan | I think rcN is really to make sure that rc1 was what we wanted | |
| 18:38:08 | superdan | mechanically it's a backport, | |
| 18:38:17 | superdan | but I can't really think the semantics can be considered the same | |
| 18:38:31 | johnthetubaguy | yeah, thats fair | |
| 18:39:14 | mriedem | so..are we saying we can backport the whole thing for rc2, or do we hold up rc1 for just the object version change and then backport the rest? | |
| 18:39:20 | johnthetubaguy | not hit an object change in an RC potential bug before, I guess it must have happened | |
| 18:39:25 | johnthetubaguy | mriedem: the first one I think | |
| 18:39:31 | mriedem | superdan: you agree? ^ | |
| 18:39:50 | superdan | mriedem: yeah, all in rc2 | |
| 18:40:08 | johnthetubaguy | its a driver specific object anyways, so the risk is lower I guess | |
| 18:40:25 | superdan | this is the thing that is not a regression in ocata though right? | |
| 18:40:47 | johnthetubaguy | it might have been newton, not sure actually | |
| 18:41:58 | EmilienM | mriedem: can you revisit https://review.openstack.org/#/c/428424/ please? | |
| 18:42:40 | mriedem | johnthetubaguy: superdan: BobBall said it was a regression in newton | |
| 18:42:48 | superdan | yeah, thought so | |
| 18:47:17 | EmilienM | mriedem: wow so https://review.openstack.org/#/c/428415/ passed for us? | |
| 18:48:01 | EmilienM | mriedem: but fail on our baremetal jobs: http://logs.openstack.org/15/428415/1/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/92a342e/console.html#_2017-02-03_01_43_02_872309 | |
| 18:48:13 | mriedem | and fails a ton of tests in nova | |
| 18:48:19 | mriedem | and we don't really know what else would break with it | |
| 18:48:23 | mriedem | since we have 0 burn in time | |
| 18:49:41 | EmilienM | mriedem: this one looks strange: http://logs.openstack.org/15/428415/1/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/92a342e/logs/undercloud/var/log/nova/nova-compute.txt.gz#_2017-02-03_01_06_02_020 | |
| 18:50:25 | mriedem | yjs | |
| 18:50:29 | mriedem | that's using ironic? | |
| 18:50:35 | EmilienM | mriedem: yes | |
| 18:52:15 | mriedem | hmm, that's odd. something not working in the resource tracking, but that stuff should come in when nova-compute starts up, | |
| 18:52:21 | mriedem | unless nova-compute is failing to get info from ironic | |
| 18:52:42 | mriedem | Total disk: 40 GB, used: 40.00 GB | |
| 18:52:44 | mriedem | right above that | |
| 18:52:48 | EmilienM | could it be nova/wsgi again? | |
| 18:53:02 | mriedem | and just before that Total disk: 40 GB, used: 0.00 GB | |
| 18:53:05 | superdan | mriedem: unless nova-compute starts before any ironic nodes are provisioned | |
| 18:53:19 | superdan | mriedem: also, they stop reporting resources for nodes that are in the middle of a change | |
| 18:53:29 | superdan | mriedem: so they will go in and out and in and out over the course of a normal lifetime | |
| 18:53:32 | superdan | it's weird | |
| 18:53:34 | EmilienM | superdan: in the case opposite happens, nova-ocmpute is able to update nodes resources? | |
| 18:53:34 | mriedem | there is an instance created right before that one that claims all the disk http://logs.openstack.org/15/428415/1/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/92a342e/logs/undercloud/var/log/nova/nova-compute.txt.gz#_2017-02-03_01_06_01_052 | |
| 18:53:40 | superdan | but jroll and I were looking at that the other day | |
| 18:54:20 | EmilienM | let me look in logstash if tripleo ci had this error before | |
| 18:54:43 | mriedem | EmilienM: logstash is super backed up | |
| 18:54:46 | jroll | I didn't do it | |
| 18:54:46 | mriedem | so probably no results there | |
| 18:55:29 | mriedem | plus and http://logs.openstack.org/15/428415/1/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/92a342e/logs/undercloud/var/log/nova/nova-compute.txt.gz#_2017-02-03_01_06_02_020 is debug level which we don't index | |
| 18:56:05 | EmilienM | mriedem: over the last 30 days, one occurence, on 427536 changfe | |
| 18:56:15 | mriedem | we only store 10 days of logs | |
| 18:56:20 | mriedem | and it's been busted for several days | |
| 18:56:22 | EmilienM | mriedem: I think it's pretty clear this patch is responsible | |
| 18:56:26 | EmilienM | even 10 days | |
| 18:56:54 | mriedem | well the patch is https://review.openstack.org/#/c/427536/ | |
| 18:56:58 | EmilienM | mriedem: http://tripleo.org/cistatus.html we run so many jobs per day, we would have seen it before | |
| 18:56:59 | mriedem | 2 deps | |
| 18:57:03 | mriedem | i don't see how https://review.openstack.org/#/c/427782/ would cause this | |
| 18:57:11 | mriedem | so what does https://review.openstack.org/#/c/427534/ do? | |
| 18:57:21 | mriedem | my money is on ^ | |
| 18:57:28 | mriedem | "Ensure that the ironic nodes have been picked up by the nova resource tracker before running nova-manage cell_v2 host discovery." | |
| 18:57:46 | EmilienM | owalsh: you around? ^ | |
| 18:57:47 | superdan | 427782 is only about getting instances via the api, really nothing to do with any of that | |
| 18:57:53 | mriedem | right | |
| 18:58:07 | mriedem | the latter is specifically about when to discover ironic hosts for resource tracking | |
| 18:58:24 | superdan | mriedem: right, and it is important because they weren't enrolling nodes by the time they ran discover | |
| 18:59:02 | superdan | without it they're 100% fail because they never have host mappings | |
| 18:59:16 | mriedem | even though, regardless of all those changes, | |
| 18:59:16 | mriedem | http://logs.openstack.org/15/428415/1/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/92a342e/logs/undercloud/var/log/nova/nova-compute.txt.gz#_2017-02-03_01_06_01_049 | |
| 18:59:24 | mriedem | Attempting claim: memory 4096 MB, disk 40 GB, vcpus 1 CPU | |
| 18:59:29 | openstackgerrit | John Perkins proposed openstack/nova master: Integrate oslo.config validator https://review.openstack.org/428974 | |
| 18:59:32 | mriedem | Total disk: 40 GB, used: 0.00 GB | |
| 18:59:39 | mriedem | so that instance claims all of the disk | |
| 18:59:46 | mriedem | which is why the next one fails | |
| 18:59:54 | mriedem | unless, there should be more nodes reporting in | |
| 19:00:09 | mriedem | i.e. we should have more than 40GB total | |
| 19:00:09 | superdan | or the scheduler raced | |
| 19:00:59 | cdent | whose 40GB is that? Is there a chance ironic is now trying to claim the wrong disk? (we are talking ironic here aren't we?) | |
| 19:01:18 | mriedem | yes ironic | |
| 19:03:02 | mriedem | 2017-02-03 01:05:33.326 28176 DEBUG nova.virt.ironic.driver [req-4a6f8f71-565c-4379-8288-347b8e9da028 - - - - -] Returning 3 available node(s) get_available_nodes /usr/lib/python2.7/site-packages/nova/virt/ironic/driver.py:610 | |
| 19:03:26 | mriedem | Hypervisor/Node resource view: name=7c2a1870-c2f8-46d5-b1a4-c72443bfc82f free_ram=8192MB free_disk=40GB free_vcpus=4 pci_devices=None | |
| 19:03:54 | mriedem | Final resource view: name=7c2a1870-c2f8-46d5-b1a4-c72443bfc82f phys_ram=8192MB used_ram=0MB phys_disk=40GB used_disk=0GB total_vcpus=4 used_vcpus=0 pci_stats=[] | |
| 19:04:05 | mriedem | Final resource view: name=aa97370e-02a2-4aa9-8833-616d30a4d589 phys_ram=8192MB used_ram=0MB phys_disk=40GB used_disk=0GB total_vcpus=4 used_vcpus=0 pci_stats=[] | |
| 19:04:13 | mriedem | Final resource view: name=6a085a83-e051-480b-bf37-d092d7bbf13e phys_ram=8192MB used_ram=0MB phys_disk=40GB used_disk=0GB total_vcpus=4 used_vcpus=0 pci_stats=[] | |
| 19:04:18 | mriedem | so those are the 3 nodes | |
| 19:04:22 | mriedem | 40GB each | |
| 19:04:36 | superdan | right and ironic doesn't sum the disks I think | |
| 19:04:51 | superdan | to prevent you from trying to schedule a 120G instance to that compute right/ | |
| 19:04:54 | mriedem | so 1 instance should be placed on 1 node right? | |
| 19:05:11 | superdan | yep | |
| 19:05:25 | superdan | I might be thinking of the summing in the api bit | |
| 19:05:31 | superdan | anyway, three nodes, 40g each | |
| 19:05:31 | mriedem | would be nice if the "Claim successful" message told us which node the resources were claimed on | |
| 19:05:43 | mriedem | in this log it's building a few instances | |