| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-06 | |||
| 19:04:29 | odyssey4me | oh hang on, I think it mighth be sha pinned | |
| 19:04:45 | mriedem | there was only one nova_api db change in newton before eol https://github.com/openstack/nova/blob/newton-eol/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/022_request_specs_spec_mediumtext.py | |
| 19:04:53 | mriedem | and shouldn't have anything to do with what you're seeing | |
| 19:05:04 | odyssey4me | heh, that makes it even weirder that this failed: https://github.com/openstack/openstack-ansible-tests/blob/stable/newton/test-vars.yml#L157 | |
| 19:05:35 | dansmith | kashyap: yeah that's why I said it didn't seem like it would :) | |
| 19:06:58 | mriedem | odyssey4me: so everything in that console output looks ok to me, | |
| 19:07:10 | mriedem | where do instances actually get created in newton before the upgrade is started? | |
| 19:07:14 | dansmith | kashyap: I was also really looking for you to comment on the potential race with the setting of the bandwidth limit in two places.. did you look at that at all? | |
| 19:07:29 | mriedem | odyssey4me: i.e. how do you know that you've actually got instances from newton to map when map_instances runs? | |
| 19:07:36 | kashyap | dansmith: No, not yet. After I finish dinner | |
| 19:08:33 | odyssey4me | mriedem by 'instances' do you mean the compute host? The terminology is confusing to me here... because in this CI run no instance (by this I mean a cloud instance, not a nova hypervisor) is created until *after* the upgrade when tempest executes. | |
| 19:09:41 | mriedem | instances == vms | |
| 19:09:52 | mriedem | odyssey4me: ok then map_instances isn't going to map anything :) | |
| 19:10:02 | mriedem | map_instances means create nova_api.instance_mappings records in the db | |
| 19:10:07 | mriedem | based on VMs in the 'nova' db | |
| 19:10:11 | mriedem | the 'nova' db is the cell1 db | |
| 19:11:08 | mriedem | odyssey4me: so looking at https://pastebin.com/C9ji6vdS again, when/how did either of those instances get created? | |
| 19:11:13 | odyssey4me | ok, btw if we're doing something dumb here then please feel free to say so with any improvement suggestions :) | |
| 19:11:13 | mriedem | the ones passed to verify_instance | |
| 19:11:23 | mriedem | odyssey4me: the ansible tasks all look fine | |
| 19:11:39 | mriedem | the ansible is like a step for step copy of what's in the install guide here https://docs.openstack.org/nova/latest/user/cells.html | |
| 19:11:59 | odyssey4me | ok, given that I did those commands after the test failed, those instances would have been created by tempest after the upgrade completed | |
| 19:12:18 | odyssey4me | the one thing that may not have run here is the online migrations | |
| 19:12:26 | dansmith | kashyap: okay thanks | |
| 19:12:42 | mriedem | odyssey4me: do you see any errors in the nova-api logs? | |
| 19:12:46 | mriedem | or nova-conductor? | |
| 19:13:24 | mriedem | also, based on that git hash for nova, the CI isn't picking up any bug fixes since the newton GA | |
| 19:13:26 | mriedem | which seems less than ideal | |
| 19:16:24 | odyssey4me | nova-conductor: No host-to-cell mapping found for selected host ocata-nova1. Setup is incomplete. | |
| 19:16:42 | odyssey4me | Failed to compute_task_build_instances: Host 'ocata-nova1' is not mapped to any cell | |
| 19:17:48 | hrw | https://marcin.juszkiewicz.com.pl/2018/02/06/graphical-console-in-openstack-aarch64/ - please read ;) | |
| 19:24:15 | tssurya | odyssey4me: can you check the value of the "mapped" column inside compute_nodes table of the nova db (cell1 db), ? if you have this host there, then there should be a record | |
| 19:26:05 | odyssey4me | hmm, select mapped from compute_nodes; gives me an unknown column error | |
| 19:26:50 | odyssey4me | yup, none of those are working | |
| 19:27:02 | odyssey4me | none of the db's have that column | |
| 19:29:49 | tssurya | there should be a table called compute_nodes only in your nova db | |
| 19:29:52 | tssurya | not in the api | |
| 19:30:41 | tssurya | odyssey4me: that is in the cell1's db | |
| 19:31:02 | odyssey4me | yup, the nova db has that table - but the table has no 'mapped' column | |
| 19:32:26 | odyssey4me | tssurya here're the columns present: https://pastebin.com/grmGWEQe | |
| 19:33:11 | odyssey4me | tssurya in case you missed it, this is a newton build (with no cells) being upgraded to an ocata build (with cells being setup) | |
| 19:33:20 | odyssey4me | cells v2 to be clear | |
| 19:35:20 | tssurya | odyssey4me: oh okay, ocata.. | |
| 19:35:49 | odyssey4me | tssurya yep :) ye olde crusty stable code ;) | |
| 19:37:02 | odyssey4me | mriedem any further thoughts or ideas? if not I'll work on fixing up the log capturing so that I can point you at a proper set of logs to peruse | |
| 19:37:54 | mriedem | odyssey4me: sorry was eating lunch, | |
| 19:38:01 | mriedem | i think the compute_nodes.mapped column was added in ocata | |
| 19:38:31 | mriedem | the host 'ocata-nova1' is a problem i think, since an earlier paste showed it was discovered in 2 different cell mappings | |
| 19:38:47 | mriedem | https://pastebin.com/mSwpZybQ | |
| 19:39:13 | mriedem | at this point i'd probably need the logs from an untouched env | |
| 19:39:38 | odyssey4me | alright, thanks much for your time and input so far | |
| 19:39:41 | mriedem | yw | |
| 19:40:21 | odyssey4me | I'll work on getting the log collection fixed up so that we can debug better. | |
| 19:44:30 | efried | mriedem: I'm going to put up three pairs of patches to dig a bit deeper, see which part of ServiceTokenAuthWrapper is actually busted. | |
| 19:44:36 | efried | mriedem: Unless you have some other plan. | |
| 19:44:42 | mriedem | so 6 patches? | |
| 19:45:04 | mriedem | i don't, no :) | |
| 19:45:21 | mriedem | we might have to put this in the release notes as a known issue | |
| 19:45:46 | efried | Yeah, going to pass through each of the service_auth-y bits of ServiceTokenAuthWrapper - which I have to do in ksa - and then a blank Nova patch for each that Depends-On its ksa buddy. | |
| 19:45:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Create volume attachment during boot from volume in compute https://review.openstack.org/541420 | |
| 19:45:53 | mriedem | ildikov: ^ cleans up that bfv legacy attach stuff | |
| 19:47:10 | ildikov | mriedem: tnx, looking | |
| 19:47:36 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: debug nova-api service version check during boot from volume https://review.openstack.org/540992 | |
| 19:51:04 | mriedem | holy f 679 check queue length | |
| 19:52:21 | melwitt | I saw there was a status message about zuul from 02:30 having problems | |
| 19:52:51 | melwitt | "[02:30:17] -openstackstatus-NOTICE: Our Zuul infrastructure is currently experiencing some problems and processing jobs very slowly, we're investigating. Please do not approve or recheck changes for now." | |
| 19:53:00 | mriedem | yeah i knew about that | |
| 19:53:22 | mriedem | melwitt: you might want to weigh in on this https://review.openstack.org/#/c/532361/ | |
| 19:53:37 | mriedem | given you'll be the ptl of the project that ruined removing mox in rocky for all projects | |
| 19:53:48 | melwitt | -_- | |
| 19:54:13 | mriedem | as the lame duck ptl, i can only fire shots across the bow | |
| 19:56:02 | mriedem | bauzas: don't forget https://review.openstack.org/#/c/526095/ | |
| 19:56:03 | cdent | melwitt, mriedem : I recommend either lying ("yeah, sure we can will do it") or just do it | |
| 19:56:24 | artom | Strictly speaking, *removing* mox is easy | |
| 19:56:25 | mriedem | cdent: i assumed we'd do what we have done in previous releases, | |
| 19:56:33 | artom | *Replacing* it with mock, OTOH... | |
| 19:56:34 | mriedem | which is we can work on it as a low priority thing | |
| 19:56:48 | mriedem | so it would be what doug said, which is forward progress | |
| 19:57:56 | cdent | mriedem: why not, since it is a goal™, make it higher priority and use the fact that it is a goal as an excuse to de-prioritize other stuff? | |
| 19:58:43 | mriedem | i don't consider it high priority | |
| 19:59:11 | cdent | isn't that what "openstack wide goals" means? | |
| 19:59:13 | melwitt | yeah. while I agree it's a fine thing to do, it's not trivial to review and takes away from reviewing important things like bug fixes and placement work that our users want and need | |
| 19:59:45 | cdent | (to be clear, I'm not too wed to the mox thing, but I do think that the goals process should mean _something_) | |
| 19:59:48 | melwitt | and I personally don't get the argument of "maintaining mox3" because, what is there to maintain? | |
| 20:00:34 | mriedem | right, i don't think 'retiring mox3' is a high priority | |
| 20:00:37 | openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Debug nova-next failure: user headers https://review.openstack.org/541427 | |
| 20:00:48 | cdent | melwitt: that seems a far more valid response to the goal process | |
| 20:00:54 | mriedem | ffs, sqlalchemy-migrate was forked over for openstack usage, and we're still using that gd thing forever | |
| 20:01:07 | cdent | much to the chagrin of zzzeek | |
| 20:01:10 | mriedem | this would be like a goal to migrate every project from sqla-migrate to that other thing | |
| 20:01:17 | mriedem | alembic | |
| 20:01:29 | mriedem | which is arguably a more useful goal than removing mox tests | |
| 20:02:17 | cdent | perhaps one of you should respond to zane with the real reasons you don't want to do it? | |
| 20:02:33 | dansmith | didn't mriedem already do that? | |
| 20:03:03 | cdent | dansmith: I'm not entirely sure. I just read his most recent response and that doesn't seem to. | |
| 20:03:16 | mriedem | i did, | |
| 20:03:18 | dansmith | wasn't his first resposne "because we can't possibly get it done" ? | |
| 20:03:24 | openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Debug nova-next failure: invalidate https://review.openstack.org/541430 | |
| 20:03:26 | cdent | But, dansmith, I was just trying to say "instead of talking about it here, maybe talk about it on the review" | |
| 20:03:32 | mriedem | i left comments on the review | |