| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-03 | |||
| 17:54:47 | openstackgerrit | Anusha Unnam proposed openstack/nova master: Reno for additional-notification-fields-for-searchlight https://review.openstack.org/428835 | |
| 18:00:34 | sc68cal | EmilienM: so looks like we had a netsplit | |
| 18:00:43 | sc68cal | EmilienM: where did I lose you | |
| 18:01:17 | EmilienM | sc68cal: better: http://logs.openstack.org/86/428786/3/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/d244aec/logs/nova/nova-api.txt.gz#_2017-02-03_17_44_34_097 | |
| 18:01:35 | EmilienM | sc68cal: so using ::1 works :-/ | |
| 18:02:39 | sc68cal | EmilienM: yep..... we have same issue with mysql vs alembic/sqlchemy | |
| 18:02:45 | aunnam | gibi, if possible can you review this patch https://review.openstack.org/#/c/428835/ | |
| 18:03:23 | sc68cal | EmilienM: https://github.com/openstack-dev/devstack/blob/master/lib/database#L77 | |
| 18:05:29 | openstackgerrit | Bob Ball proposed openstack/nova master: Fix live migrate with XenServer https://review.openstack.org/424428 | |
| 18:05:31 | BobBall | johnthetubaguy: ^^ Hopefully that'll fix the different network concern | |
| 18:09:43 | cdent | mriedem, johnthetubaguy: the tagging has not happened has it? | |
| 18:10:04 | cdent | writing up some notes and wanted to know if https://review.openstack.org/#/c/428375/ was part of the now-ish release | |
| 18:11:07 | johnthetubaguy | https://review.openstack.org/#/c/428531/ says we haven't tagged yet, looks like we can include the ironic one if we wish | |
| 18:26:03 | mriedem | cdent: not yet | |
| 18:26:25 | mriedem | but now that https://review.openstack.org/#/c/428375/ is merged and i'm back from lunch, i'm going to rev the rc1 tag patch | |
| 18:27:35 | cdent | mriedem: how very exciting | |
| 18:31:05 | mriedem | i know, i'm cutting glass over here | |
| 18:33:02 | superdan | lol | |
| 18:33:03 | openstackgerrit | Sujitha proposed openstack/nova master: Reno for additional-notification-fields-for-searchlight https://review.openstack.org/428835 | |
| 18:34:33 | johnthetubaguy | mriedem: I am guessing there is no way to backport this one, if it misses RC1 right? https://review.openstack.org/#/c/424428 | |
| 18:35:44 | mriedem | johnthetubaguy: not with the object version change, | |
| 18:35:46 | mriedem | but there is rc2 | |
| 18:35:56 | johnthetubaguy | mriedem: what if we just get the object change in? | |
| 18:36:04 | mriedem | heh, so i thought about that yesterday | |
| 18:36:13 | johnthetubaguy | mriedem: but rc2 is a backport I guess | |
| 18:36:22 | mriedem | yes rc2 is a backport to stable/ocata technically | |
| 18:36:35 | mriedem | but it'd also be pre-release | |
| 18:36:36 | superdan | johnthetubaguy: mriedem it's okay if the versions have no gap, and I think this is sufficiently defensive | |
| 18:36:47 | superdan | I mean, it's technically okay if there's no gap | |
| 18:37:20 | johnthetubaguy | superdan: oh, true, the likely hood of moving from stable to a commit without that commit seems fairly tiny | |
| 18:37:46 | superdan | johnthetubaguy: the biggestdeal is if you happen to get conductor on rc1 and compute on rc2 | |
| 18:37:50 | superdan | however, | |
| 18:38:03 | superdan | I think rcN is really to make sure that rc1 was what we wanted | |
| 18:38:08 | superdan | mechanically it's a backport, | |
| 18:38:17 | superdan | but I can't really think the semantics can be considered the same | |
| 18:38:31 | johnthetubaguy | yeah, thats fair | |
| 18:39:14 | mriedem | so..are we saying we can backport the whole thing for rc2, or do we hold up rc1 for just the object version change and then backport the rest? | |
| 18:39:20 | johnthetubaguy | not hit an object change in an RC potential bug before, I guess it must have happened | |
| 18:39:25 | johnthetubaguy | mriedem: the first one I think | |
| 18:39:31 | mriedem | superdan: you agree? ^ | |
| 18:39:50 | superdan | mriedem: yeah, all in rc2 | |
| 18:40:08 | johnthetubaguy | its a driver specific object anyways, so the risk is lower I guess | |
| 18:40:25 | superdan | this is the thing that is not a regression in ocata though right? | |
| 18:40:47 | johnthetubaguy | it might have been newton, not sure actually | |
| 18:41:58 | EmilienM | mriedem: can you revisit https://review.openstack.org/#/c/428424/ please? | |
| 18:42:40 | mriedem | johnthetubaguy: superdan: BobBall said it was a regression in newton | |
| 18:42:48 | superdan | yeah, thought so | |
| 18:47:17 | EmilienM | mriedem: wow so https://review.openstack.org/#/c/428415/ passed for us? | |
| 18:48:01 | EmilienM | mriedem: but fail on our baremetal jobs: http://logs.openstack.org/15/428415/1/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/92a342e/console.html#_2017-02-03_01_43_02_872309 | |
| 18:48:13 | mriedem | and fails a ton of tests in nova | |
| 18:48:19 | mriedem | and we don't really know what else would break with it | |
| 18:48:23 | mriedem | since we have 0 burn in time | |
| 18:49:41 | EmilienM | mriedem: this one looks strange: http://logs.openstack.org/15/428415/1/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/92a342e/logs/undercloud/var/log/nova/nova-compute.txt.gz#_2017-02-03_01_06_02_020 | |
| 18:50:25 | mriedem | yjs | |
| 18:50:29 | mriedem | that's using ironic? | |
| 18:50:35 | EmilienM | mriedem: yes | |
| 18:52:15 | mriedem | hmm, that's odd. something not working in the resource tracking, but that stuff should come in when nova-compute starts up, | |
| 18:52:21 | mriedem | unless nova-compute is failing to get info from ironic | |
| 18:52:42 | mriedem | Total disk: 40 GB, used: 40.00 GB | |
| 18:52:44 | mriedem | right above that | |
| 18:52:48 | EmilienM | could it be nova/wsgi again? | |
| 18:53:02 | mriedem | and just before that Total disk: 40 GB, used: 0.00 GB | |
| 18:53:05 | superdan | mriedem: unless nova-compute starts before any ironic nodes are provisioned | |
| 18:53:19 | superdan | mriedem: also, they stop reporting resources for nodes that are in the middle of a change | |
| 18:53:29 | superdan | mriedem: so they will go in and out and in and out over the course of a normal lifetime | |
| 18:53:32 | superdan | it's weird | |
| 18:53:34 | EmilienM | superdan: in the case opposite happens, nova-ocmpute is able to update nodes resources? | |
| 18:53:34 | mriedem | there is an instance created right before that one that claims all the disk http://logs.openstack.org/15/428415/1/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/92a342e/logs/undercloud/var/log/nova/nova-compute.txt.gz#_2017-02-03_01_06_01_052 | |
| 18:53:40 | superdan | but jroll and I were looking at that the other day | |
| 18:54:20 | EmilienM | let me look in logstash if tripleo ci had this error before | |
| 18:54:43 | mriedem | EmilienM: logstash is super backed up | |
| 18:54:46 | jroll | I didn't do it | |
| 18:54:46 | mriedem | so probably no results there | |
| 18:55:29 | mriedem | plus and http://logs.openstack.org/15/428415/1/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/92a342e/logs/undercloud/var/log/nova/nova-compute.txt.gz#_2017-02-03_01_06_02_020 is debug level which we don't index | |
| 18:56:05 | EmilienM | mriedem: over the last 30 days, one occurence, on 427536 changfe | |
| 18:56:15 | mriedem | we only store 10 days of logs | |
| 18:56:20 | mriedem | and it's been busted for several days | |
| 18:56:22 | EmilienM | mriedem: I think it's pretty clear this patch is responsible | |
| 18:56:26 | EmilienM | even 10 days | |
| 18:56:54 | mriedem | well the patch is https://review.openstack.org/#/c/427536/ | |
| 18:56:58 | EmilienM | mriedem: http://tripleo.org/cistatus.html we run so many jobs per day, we would have seen it before | |
| 18:56:59 | mriedem | 2 deps | |
| 18:57:03 | mriedem | i don't see how https://review.openstack.org/#/c/427782/ would cause this | |
| 18:57:11 | mriedem | so what does https://review.openstack.org/#/c/427534/ do? | |
| 18:57:21 | mriedem | my money is on ^ | |
| 18:57:28 | mriedem | "Ensure that the ironic nodes have been picked up by the nova resource tracker before running nova-manage cell_v2 host discovery." | |
| 18:57:46 | EmilienM | owalsh: you around? ^ | |
| 18:57:47 | superdan | 427782 is only about getting instances via the api, really nothing to do with any of that | |
| 18:57:53 | mriedem | right | |
| 18:58:07 | mriedem | the latter is specifically about when to discover ironic hosts for resource tracking | |
| 18:58:24 | superdan | mriedem: right, and it is important because they weren't enrolling nodes by the time they ran discover | |
| 18:59:02 | superdan | without it they're 100% fail because they never have host mappings | |
| 18:59:16 | mriedem | even though, regardless of all those changes, | |
| 18:59:16 | mriedem | http://logs.openstack.org/15/428415/1/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/92a342e/logs/undercloud/var/log/nova/nova-compute.txt.gz#_2017-02-03_01_06_01_049 | |
| 18:59:24 | mriedem | Attempting claim: memory 4096 MB, disk 40 GB, vcpus 1 CPU | |
| 18:59:29 | openstackgerrit | John Perkins proposed openstack/nova master: Integrate oslo.config validator https://review.openstack.org/428974 | |
| 18:59:32 | mriedem | Total disk: 40 GB, used: 0.00 GB | |
| 18:59:39 | mriedem | so that instance claims all of the disk | |
| 18:59:46 | mriedem | which is why the next one fails | |
| 18:59:54 | mriedem | unless, there should be more nodes reporting in | |